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Abstract 

National  strategic-level  guidance  has  outlined  how  the  U.S.  will  continue  in  “The 
Long  War.”  From  the  President  downward,  emphasis  has  been  placed  on  bolstering  U.S. 
national  security  interests  by  working  with  and  through  foreign  counterparts.  The  U.S. 
Air  Force  has  assumed  the  leadership  role  to  develop  the  aviation  capacity  and  capability 
of  key  partner  nations  (PN).  The  key  asset  to  accomplishing  this  objective  is  to  use  Air 
Advisors  (AA).  These  AAs  are  trained  to  assess,  advise,  train,  assist,  and  equip  selected 
PNs.  As  the  worldwide  leader  in  air  mobility  capability,  Air  Mobility  Command  has 
activated  two  Mobility  Support  Advisory  Squadrons  (MSAS)  to  manage  the  AAs  and 
their  taskings. 

While  initial,  broad-based  guidance  was  provided,  no  direction  was  given  on 
how  the  MSAS  are  to  measure  and  report  their  effectiveness  at  conducting  the  Building 
Partner  Capacity  (BPC)  mission.  Since  the  MSAS  seek  to  establish,  build,  and  foster 
relationships,  this  process  is  time-consuming  and  lengthy  in  nature.  Furthermore,  it  is 
difficult  to  quantify  expected  outcomes  such  as  trust,  good-will,  and  positive  feelings 
towards  the  U.S. — thus  making  measurement  of  outcome  almost  entirely  subjective. 

This  qualitative  research  effort  utilized  a  Delphi  study  to  assemble  20  expert 
panelists  to  detennine  the  key  criteria  the  MSAS  could  use  to  measure,  track,  and  report 
unit  effectiveness  toward  accomplishing  their  BPC  mission.  The  data  obtained  through 
this  study  helped  to  fonnulate  recommendations  that  could  be  utilized  by  Air  Force 
decision  makers  at  the  tactical,  operational,  and  strategic  levels  on  how  to  shape  the 
future  of  PN  engagements  made  through  the  BPC  initiative. 


IV 


Acknowledgments 


Col  Christopher  Pike,  former  Vice-Commander,  2 1st  Expeditionary  Mobility 
Task  Force,  was  the  study  sponsor.  I  greatly  appreciate  Col  Pike’s  enthusiastic  support 
and  assistance  with  keeping  the  research  properly  focused  and  desire  to  make  this  effort 
count. 

Dr.  Alan  Heminger,  AFIT/ENV,  was  the  study  advisor.  After  hearing  my 
research  topic  proposal,  it  was  Dr.  Heminger  that  recommended  using  the  Delphi  method 
to  answer  my  research  questions.  Throughout  the  research  effort,  he  provided  me  timely 
and  value-added  guidance  which  ensured  I  was  able  to  get  the  most  out  of  the  Delphi. 

I’m  truly  grateful  for  his  passion  for  teaching  and  student  learning. 

Thanks  to  Pamela  Bennet  Bardot,  IJSAF  Expeditionary  Center  Librarian- 
Extraordinaire  for  graciously  entertaining  my  numerous  requests  for  information  and 
helping  me  get  smarter  on  the  Delphi  method. 

Thanks  to  Lt  Col  Thomas  Adkins,  Commander,  818  MSAS  and  Lt  Col  Joseph 
“Nacho”  Sanchez,  Commander,  571  MSAS,  for  openly  sharing  their  perspective,  insight, 
and  personal  experiences  as  they  make  history  serving  as  the  first  two  MSAS 
commanders. 

Thanks  to  Maj  Phil  Smith,  818  MSAS/DO,  for  providing  the  idea  that  ultimately 
became  the  topic  of  this  research  paper. 

Thanks  to  Hale  Laughlin,  AFSOC/A8,  for  taking  his  time  to  share  his  wealth  of 
knowledge  and  experience  on  the  Building  Partnerships/Building  Partner  Capacity 
(BP/BPC)  issue  as  well  as  point  me  to  other  valuable  sources  of  infonnation. 


v 


Thanks  to  Col  J.  Olaf  Holm,  Commandant,  Air  Advisor  Academy  (AAA)  and  the 


AAA  cadre  for  allowing  me  to  sit  in  their  classes  and  share  their  outlook. 

Thanks  to  Col  Eric  Watkins,  HAF/A3;  Ken  Arteaga,  AETC/A3;  and  Deo 
Lachman  and  Paul  Judge,  AMC/A3;  for  keeping  me  infonned  as  developments  occurred 
within  the  BP/BPC  program. 

Thanks  to  the  Delphi  panel  for  your  patience,  time,  and  effort  spent  sharing  your 
personal,  honest  perspective  on  the  BP/BPC  mission.  This  made  the  research  study  an 
overwhelming  success.  Without  a  doubt,  you  all  are  champions  for  the  cause.  The 
members  are  as  follows: 


Lt  Gen  Robert  Allardice 

AMC/CV 

Lt  Col  Ernest  Teichert 

NDU  Student 

Col  John  Caimey 

621  CRW/CV 

Maj  Wes  Eagle 

87  LRS/CC 

Col  Javier  Delucca 

571  CRG/CD 

MSgt  Sameul  Nowman 

439  AEAS 

Col  Valentine  Dugie 

SAL/1A 

MSgt  Robert  Weatherly 

439  AEAS 

Col  J.  Olaf  Holm 

AAA/CC 

Mrs.  Susan  Alaniz 

AFSOS/BPACC  Director 

Col  Scott  Graham 

HAL/A3 

Mr.  Kenneth  Arteaga 

AETC/A3 

Col  Eric  Watkins 

HAL/A3 

Mr.  Paul  Judge 

AMC/A3 

Lt  Col  Thomas  Adkins,  11 

818  MSAS/CC 

Mr.  Deo  Lachman 

AMC/A3 

Lt  Col  Michael  Grub 

1  SOG/CD 

Mr.  G.  Hale  Laughlin 

AFSOC/A8 

Lt  Col  Joseph  Sanchez 

571  MSAS/CC 

Mr.  Pasquale  (Pat)  Capriglione 

Department  of  State  (ret) 

Finally,  my  immeasurable  gratitude  and  love  go  to  my  wife,  children,  and  parents. 
Thank  you  for  your  steadfast  encouragement  throughout  this  study  and  academic  year. 
Your  support  made  all  of  this  possible.  Thank  you. 


vi 


Table  of  Contents 


Page 

Abstract . iv 

Acknowledgments . v 

Table  of  Contents . vii 

List  of  Figures . ixi 

List  of  Tables . ix 

I.  Introduction . 1 

II.  Literature  Review . 4 

National  Guidance . 5 

Air  Force  Guidance . 6 

Development  of  the  Building  Partnership/Building  Partner  Capacity  Mission . 7 

Development  of  2 1st  Century  Air  Advising . 10 

Importance  of  Mobility  Ainnen  to  BP/BPC . 12 

Concept  of  Mobility  Support  Advisory  Squadron  Employment . 13 

Tasking  Process . 15 

Anticipated  Benefits . 17 

Challenges . 18 

III.  Methodology . 19 

The  Delphi  Method . 19 

IV.  Analysis  and  Results . 25 

Application  of  Methodology . 25 

Delphi  Study  Execution . 28 

Analysis  of  Results . 41 

V.  Conclusion  and  Recommendations . 47 

Conclusions  of  Research . 47 

Significance  of  Research . 49 

Noteworthy  Advancements  Towards  Developing  the  AF  BP/BPC  Program . 50 

Recommendations  for  Action . 52 

Limitations  of  the  Study . 56 

Recommendations  for  Future  Research . 58 

vii 


59 


Summary 

Appendix  A . 61 

Appendix  B . 63 

Appendix  C . 64 

Appendix  D . 69 

Appendix  E . 75 

Bibliography . 79 

Vita . 83 


viii 


List  of  Figures 


Page 


Figure  1:  Parnter  Nation  Air  Mobility  System  Development . 12 

Figure  2:  Phasing  Model . 14 

Figure  3:  Irregular  Warfare  Approaches . 15 

Figure  4:  Delphi  Strengths  and  Limitations . 23 

Figure  5:  Delphi  Questionnaire  2 . 33 

Figure  6:  Delphi  Questionnaire  3a . 37 

Figure  7:  Delphi  Questionnaire  3b . 38 


IX 


List  of  Tables 


Page 


Table  1:  Panel’s  Demographic  Detail . 29 

Table  2:  Categorized  Responses  and  Frequency . 3 1 

Table  3:  Criteria  That  Consensus  Was  Reached  On  in  Round  2 . 35 

Table  4:  Summary  of  Median  and  IQR  Scores . 40 

Table  5:  Summary  of  Participation . 41 

Table  6:  Research  Question  1  Final  Criteria . 42 

Table  7:  Research  Question  2  Final  Criteria . 43 


x 


DETERMINING  MOBILITY  SUPPORT  ADVISORY  SQUADRON  EFFECTIVENESS 
IN  SUPPORT  OF  BUILDING  PARTNER  CAPACITY 

I.  Introduction 

National  strategic-level  guidance  has  outlined  how  the  U.S.  will  continue  to 
endeavor  in  “The  Long  War.”  From  the  President  downward,  emphasis  has  been  placed 
on  bolstering  U.S.  national  security  interests  by  working  with  and  through  foreign 
counterparts.  The  U.S.  Air  Force  has  assumed  the  leadership  role  to  develop  the  aviation 
capacity  and  capability  of  selected  partner  nations  (PN).  The  key  assets  to  accomplishing 
this  objective  are  Air  Advisors  (AA).  The  Air  Force  defines  an  AA  as  “an  Ainnan 
specially  trained  and  educated  to  apply  aviation  support  and  operational  expertise  to 
assess,  train,  advise,  assist,  and  equip  PNs  in  the  development  and  application  of  their 
aviation  enterprise  to  meet  their  national  needs,  in  support  of  U.S.  interests”  (Air 
Mobility  Command,  20 1 0a:58). 

As  the  worldwide  leader  in  air  mobility  capability,  Air  Mobility  Command 
(AMC)  answered  this  charge  in  December  2010  via  the  AMC  Programming  Plan  10-01, 
AMC  Building  Partner  Capacity  Unit  Activations  for  Con  tingency  Response  Wings  at 
Travis  AFB  and  Joint  Base  McGuire-Dix-Lakehurst  (Air  Mobility  Command,  201  Ob). 
This  document  directed  the  rapid  creation  of  two  Mobility  Support  Advisory  Squadrons 
(MSAS)  and  aligned  one  under  each  of  two  U.S. -based  Contingency  Response  Wings 
(CRW).  The  MSAS  were  assigned  commanders  in  April  2011,  which  marked  their 
official  activation.  As  designed,  each  of  these  squadrons  is  assigned  specific  geographic 
areas  in  which  to  operate.  The  squadrons  will  supply  AAs  in  order  to  further  the  aviation 
enterprise  development  of  specified  PNs.  The  818th  MSAS  is  aligned  to  work  with  U.S. 
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Africa  Command  (USAFRICOM)  and  its  air  component,  3rd  Air  Force  (AFAFRICOM). 
The  571st  MSAS  is  aligned  with  U.S.  Southern  Command  (USSOUTHCOM)  and  its  air 
component  12th  Air  Force  (AFSOUTH). 

While  initial,  broad-based  guidance  was  drafted  and  published  in  order  to  assist 
timely  unit  stand-up  actions,  in-depth  guidance  necessary  for  success  is  still  being 
developed.  One  key  element  that  has  not  been  addressed  pertains  to  assessment. 
Specifically,  no  direction  was  given  on  how  the  MSAS  are  to  measure  their  effectiveness 
at  conducting  the  Building  Partner  Capacity  (BPC)  mission.  What  makes  this 
requirement  even  more  difficult  is  that  establishing,  building,  and  fostering  relationships 
with  PNs  is  time-consuming.  Furthermore,  it  is  difficult  to  quantify  expected  outcomes 
such  as  trust,  good-will,  and  positive  feelings  towards  the  U.S.  This  makes  measuring 
outcome,  at  present,  almost  entirely  subjective  in  nature. 

Therefore,  this  qualitative  research  effort  used  a  Delphi  study  to  detennine  what 
appropriate  measurement  criteria  and  reporting  mechanism  the  MSAS  could  use  to  track 
and  report  unit  effectiveness  toward  BPC.  From  the  data  collected,  analyzed,  and 
reported,  this  study  proposes  that  leadership  can  then  make  the  necessary  decisions,  at  the 
tactical,  operational,  and  strategic  levels,  on  how  to  continue  with  PN  engagements  via 
the  MSAS.  Furthermore,  a  standardized,  realistic  set  of  performance  criteria  can  ensure 
standardization  between  the  MSAS  and  give  AMC  a  baseline  from  which  to  inspect 
compliance  and  mission  readiness. 

Achieving  this  goal  will  be  difficult  due  to  the  subjectivity  and  challenge  of 
quantifying  desired  effects.  Furthennore,  the  tactical-level  actions  each  MSAS  takes 
with  their  assigned  PNs  must  be  able  to  be  conveyed  in  some  manner  that  demonstrates 
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the  effectiveness  and  progress  of  the  overall  engagement.  This  is  important  since  these 
outcomes  are  tied  to  operational  and  strategic-level  end  states  as  determined  by  AMC,  the 
geographic  combatant  commander  (GCC),  and  up  to  the  President.  If  done  correctly, 
outcomes  may  garner  tangible  results  such  as  country  access,  overfly  rights,  and  coalition 
support  during  humanitarian  assistance/disaster  response  activities.  The  data  obtained 
through  this  study  will  form  recommendations  that  will  be  offered  to  Air  Force  decision 
makers  at  the  Headquarters,  Major  Command  (e.g.  Air  Mobility  Command),  and  wing 
levels  in  order  to  shape  the  future  of  this  mission. 


3 


II.  Literature  Review 


The  documents  reviewed  for  this  study  can  be  broken  into  the  following 
categories: 

National  Guidance 

National-level  references  such  as  the  2010  National  Security  Strategy,  2008 
National  Defense  Strategy,  2011  National  Military  Strategy,  and  2006  and  2010 
Quadrennial  Defense  Review  Reports  were  referenced  to  provide  the  background  and 
overall  purpose  of  the  BP/  BPC  initiative.  Additionally,  Joint  Service  Doctrine  was  also 
studied  in  order  to  gain  further  understanding  on  how  Security  Cooperation  programs  and 
initiatives  have  been  designed  and  implemented. 

Air  Force  Guidance 

In  order  to  get  senior  Air  Force  leader  interpretation  and  subsequent  direction  on 
how  to  support  BP,  the  following  service-level  documents  were  examined:  Air  Force 
Doctrine  Document  (AFDD)  3-22,  Foreign  Internal  Defense,  AFDD  2-3,  Irregular 
Warfare,  Air  Force  Global  Partnership  Strategy,  The  21st  Century  Air  Force  Irregular 
Warfare  Strategy,  AMC  Air  Mobility  System  Building  Partnerships  Concept  of 
Employment,  and  Building  Partnership  Capacity  Trifold  Pamphlet. 

Analysis  Reports,  Research  Papers,  Journal  and  Media  Articles 

To  gain  further  understanding  of  the  topic  and  to  see  what  areas  of  BP  have 
previously  received  interest  and  study,  the  author  reviewed  several  RAND  analysis 
reports,  various  Intermediate  and  Senior  Developmental  Education  graduate  research 
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papers,  as  well  as  various  public  news  and  journal  articles.  Additionally,  non-military 
based  articles  were  referenced  to  gain  related  insight  from  the  Health  Care  community, 
religious  community,  and  other  governmental  agencies  such  as  the  Peace  Corps. 


National  Guidance 

Whenever  advisable,  the  United  States  will  work  with  or  through  others: 
enabling  allied  and  partner  capabilities,  building  their  capacity  and  developing 
collaborative  mechanisms  to  share  the  decisions,  risks  and  responsibilities  of  today ’s 
complex  challenges.  The  United  States  must  work  with  new  international  partners  in  less 
familiar  areas  of  the  world  to  reduce  the  drivers  of  instability,  prevent  terrorist  attacks  or 
disrupt  their  networks,  to  deny  sanctuary  to  terrorists  anywhere  in  the  world,  to  separate 
terrorists  from  host  populations  and  ultimately  to  defeat  them. 

-  Building  Partnership  Capacity:  QDR  Execution  Roadmap,  2006:5-6. 

From  the  strategic-level  down  to  the  tactical-level  of  our  government,  direction 

has  been  given  on  how  the  U.S.,  through  its  various  governmental  departments,  must  act 

to  bolster  national  security.  This  direction  is  abundantly  clear  within  the  Department  of 

Defense  as  evidenced  by  the  multiple  documents  published.  President  Obama,  through 

the  2010  National  Security  Strategy,  declared,  “Our  military  will  continue  strengthening 

its  capacity  to  partner  with  foreign  counterparts,  train  and  assist  security  forces,  and 

pursue  military-to-military  ties  with  a  broad  range  of  governments”  (Obama,  2010: 1 1). 

The  2008  National  Defense  Strategy  (NDS)  states: 

. .  .arguably,  the  most  important  military  component  of  the  struggle  against  violent 
extremists  is  not  the  fighting  we  do  ourselves,  but  how  well  we  help  prepare  our 
partners  to  defend  and  govern  themselves... We  will  support,  train,  advise,  and 
equip  partner  security  forces  to  counter  insurgences,  terrorism,  proliferation,  and 
other  threats.  We  will  assist  other  countries  in  improving  their  capabilities 
through  security  cooperation,  just  as  we  will  leam  valuable  skills  and  information 
from  others  better  situated  to  understand  some  of  the  complex  challenges  we  face 
together  (Department  of  Defense,  2008:8,  15-16). 
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Furthermore,  Department  of  Defense  Directive  5100.1,  Functions  of  the 
Department  of  Defense  (DOD)  and  Its  Major  Components,  states  that  shaping  activities 
and  improving  relationships  with  Partner  Nations  (PNs)  are  significant  to  the  DOD 
(Department  of  Defense,  2010a).  In  January  2012,  in  support  of  the  President’s  new 
strategic  direction  for  the  Department,  Secretary  of  Defense  Panetta  published  new 
strategic  guidance  via  the  Sustaining  U.S.  Global  Leadership:  Priorities  for  21st  Century 
Defense.  The  guidance  states  that  .  .strengthening  alliances  and  partnerships  across  all 
regions”  is  a  requirement  for  sustaining  U.S.  global  leadership  (Department  of  Defense, 
2012).  Finally,  the  2011  National  Military  Strategy  conveys,  “We  will  strengthen  and 
expand  our  network  of  partnerships  to  enable  partner  capacity  to  enhance  security.  This 
will  help  reduce  potential  safe-havens  before  violent  extremism  can  take  root” 
(Department  of  Defense,  201 1:6). 

Air  Force  Guidance 

In  support  of  this  national  focus  area  of  building  partnerships  (BP)  with  PNs,  the 
Secretary  of  the  Air  Force  (SECAF)  and  Chief  of  Staff  (CSAF)  initially  published  the  Air 
Force  Global  Partnership  Strategy:  Building  Partnerships  for  the  21st  Century  in  2008 
as  a  means  to  outline  the  service’s  approach.  In  the  Foreword  section  of  the  document, 
the  Air  Force  leadership  highlights  that  “the  USAF  must  build. .  .relationships  with 
partner  air  forces  of  all  economic  means  and  available  resources...  [as  well  as  consider] 
collaborating  with  advanced  nations  to  assist  less  capable  nations... when  mutually 
beneficial”  (Donley  and  Schwartz,  2008:i).  To  further  solidify  the  importance  of  this 
initiative,  the  SECAF  and  CSAF  via  the  2009  U.S.  Air  Force  Posture  Statement  added 
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BP  as  one  of  the  12  Air  Force  Core  Functions  (Donley  and  Schwartz,  2009).  Most 
recently,  in  recognition  of  the  need  to  adapt  to  a  rapidly  changing  strategic  environment, 
AF  leadership  published  the  2011  U.S.  Air  Force  Global  Partnership  Strategy  (AFGPS) 
to  supersede  the  2008  document.  This  latest  document  outlines  the  service’s  new  strategy 
to  apply  ends,  ways,  and  means  towards  AF  security  cooperation  and  combatant 
commander  support  efforts.  It  emphasizes,  “We  can’t  have  Global  Vigilance,  Reach,  and 
Power  for  America  without  Global  Partnerships”  (Donley  and  Schwartz,  2011).  To  that 
end,  the  U.S.  Air  Force  has  assumed  “the  leadership  role  in  developing  PN  aviation 
capacity  worldwide  in  support  of  U.S.  security  interests”  (Air  Mobility  Command, 

2010a:  1 1). 


Development  of  Building  Partnership/Building  Partner  Capacity  Mission 

BP  affords  the  Air  Force  low  cost  options  that  can  significantly  enhance  the 
security  of  the  U.S.  By  appropriately  utilizing  its  capabilities  and/or  changing  existing 
processes/procedures  the  USAF  can  optimize  its  ability  to  train  and  educate  partners. 
Air  mobility  goes  beyond  the  sum  of  its  parts  -  it  often  acts  as  the  catalyst  in  political, 
economic,  and  military  arenas.  Beginning  with  a  thorough  initial  assessment  of 
capabilities  and  needs  for  PNs,  AMC  selects  the  appropriate  ways  and  means  for 
building,  sustaining,  expanding,  and  guiding  partnerships.  Concepts  of  operations, 
interoperability,  required  training,  security,  logistics,  and  maintenance  and 
communication  infrastructure  are  necessary  to  support  AMS  development  for  PNs. 

-  Air  Mobility  Systems  Building  Partnerships  Concept  of  Employment,  2010a:  15. 

As  a  subset  of  Irregular  Warfare  (IW),  Building  Partnership  (BP)  is  the  ability  to 

set  the  conditions  for  interaction  with  partner,  competitor,  or  adversary  leaders,  military 

forces,  or  relevant  populations  by  developing  and  presenting  information  and  conducting 

activities  to  affect  their  perceptions,  will,  behavior,  and  capabilities  (Department  of  the 

Air  Force,  2010).  The  U.S.  Air  Force ’s  Concept  of  Employment  (CONEMP)  for 

Institutionalizing  Building  Partnerships  Into  Contingency  Response  Forces  further 
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defines  Building  Partnership  Capacity  (BPC),  a  subset  of  BP,  as  “the  ability  to  assist 
domestic  and  foreign  partners  and  institutions  with  the  development  of  their  capabilities 
and  capacities — for  mutual  benefit — to  address  U.S.,  national  or  shared  global  security 
interests”  (Department  of  the  Air  Force,  2010:3). 

Therefore,  the  AF  goal  is  to  assist  PNs  to  create  an  aviation  enterprise — the  total 
of  a  nation’s  aviation  capability  and  capacity — that  contributes  to  its  security,  its 
government’s  legitimacy  and  stability,  and  its  ability  to  combat  terrorist  networks,  drug 
cartel,  and  criminal  organizations  in  order  to  prevent  lethal  threats  from  originating 
within  weak  or  struggling  states  (Livingston,  2011).  To  accomplish  this,  the  USAF 
Global  Partnership  Strategy  (Donley  and  Schwartz,  2008)  directed  the  AF  BP  program 
to  target  four  strategic  end  states: 

•  Establish,  sustain,  and  expand  global  partnerships  that  are  mutually  beneficial. 

•  Provide  global  partners  the  capability  and  capacity  necessary  to  provide  for  their 
own  national  security. 

•  Establish  the  capacity  to  train,  advise,  and  assist  foreign  air  forces,  while 
conducting  partnership  activities  using  USAF  Airmen  with  the  appropriate 
language  and  cultural  skills. 

•  Develop  and  enhance  partnership  capabilities  to  ensure  interoperability, 
integration,  and  interdependence,  as  appropriate. 

These  key  end  states,  as  highlighted  in  this  document,  emphasize  the  fact  that 
“...aviation  is  a  strategic  asset  to  all  sovereign  nations”  (Donley  and  Schwartz,  2008:2). 
This  versatile  capability  provides  benefits  to  both  military  and  civilian  uses. 

Furthermore,  aviation  resources  can  accomplish  critical  roles  for  political  administration, 
security,  and  social  needs  during  humanitarian  or  natural  disasters  as  well  as  contribute  to 
the  economic  growth  and  development  of  a  nation  (Donley  and  Schwartz,  2008).  To 
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further  this  proposition,  the  CSAF  assembled  the  Irregular  Warfare  Tiger  Team  (IWTT) 
in  2009  to  “determine  the  U.S.  Air  Force’s  irregular  warfare  (IW),  building  partnerships 
(BP),  and  air  advisor  (AA)  requirements,  gaps,  and  shortfalls”  (Department  of  the  Air 
Force,  2009: 1).  During  April  2009,  two  teams  traveled  to  all  the  air  components  and 
conducted  interviews  with  the  people  responsible  for  satisfying  U.S.  Air  Force 
component  requirements  for  IW  and  BP.  The  Tiger  Team’s  tasks  were  to  provide 
doctrine,  organization,  training,  materiel,  leadership,  personnel,  and  facilities 
(DOTMLPF)  solutions  within  the  USAF  to  meet  combatant  commander  (CCDR) 
demands  and  to  make  recommendations  for  incorporating  IW,  BP,  and  AA  requirements 
into  the  component’s  designed  operational  capability  (DOC)  statements  (Department  of 
the  Air  Force,  2009).  The  underlying  premise  that  guided  the  team  was  the  ardent  belief 
that,  “the  security,  stability,  and  economic  development  of  a  nation  in  the  early  2 1  st 
century  are  inextricably  linked  to  its  aviation  resource  capacity  and  capability” 
(Department  of  the  Air  Force,  2009:ii). 

Today,  the  2011  AFGPS  highlights  that  the  “USAF  must  actively  partner  with  the 
global  community  of  airmen  to  further  U.S.  and  partner  nation  mutual  interests  in  air, 
space,  and  cyberspace. . .  the  USAF  acknowledges  that  it  must  not  limit  itself  to  the 
relationships  of  the  past  and  must  broaden  its  scope  to  include  partnerships  for  new 
situations  and  circumstances”  (Donley  and  Schwartz,  201 1:6).  In  order  to  accomplish 
this  task,  Donley  and  Schwartz  (2011)  have  tasked  the  AF  to  focus  security  cooperation 
efforts  to  build  the  capabilities  of  at-risk  and  underdeveloped  PNs  so  that  they  are  able  to 
defend  themselves  against  the  threats  of  today  and  tomorrow.  Furthermore,  “. .  .all 
military  and  defense  related  activities  with  a  foreign  partner  can  range  from  efforts 
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generating  goodwill  and  access  to  assisting  partner  nations  in  building  their  capabilities  to 
defend  themselves”  (Donley  and  Schwartz,  2011 :6).  Specifically,  in  accordance  with  the 
2011 AFGSP  (2011),  the  USAF  seeks  to: 

•  Employ  USAF  security  cooperation  activities  in  support  of  coalition  efforts  to 
counter  violent  extremism. 

•  Collaborate  with  partner  nation  Air  Forces  to  deter  and  defeat  aggression. 

•  Strengthen  international  and  regional  security. 

•  Shape  the  future  coalition  Air  Force. 

While  these  outcomes  are  the  desired  end-states  for  AF  security  cooperation 
efforts,  the  ways  or  methods  to  achieve  them  are  through  a  variety  of  military 
capabilities.  Air  Advisors  embody  a  versatile  option  that  can  be  utilized  to  deliver  key 
AF  core  functions. 


Development  of  21st  Century  Air  Advising 

[T]he  Department  must  be  prepared  to  grow  a  new  team  of 
leaders  and  operators,  who  are  comfortable  working  in  remote  regions  of  the  world, 
dealing  with  local  and  tribal  communities,  adapting  to  foreign  languages  and  cultures, 
working  with  local  networks,  operating  alongside  or  within  United  Nations 
organizations,  and  working  alongside  non-governmental  organizations  to  further  U.S. 
and  partner  interests  through  personal  engagement,  persuasion  and  quiet  influence  - 
rather  than  through  military  force  alone. 

-  Building  Partnership  Capacity:  QDR  Execution  Roadmap,  2006:6. 

Air  Advisors  (AA)  are  the  key  asset  to  accomplishing  the  aforementioned 

objective.  The  USAF  Air  Advising  Operating  Concept  defines  an  AA  as  “An 

Ainnan  specially  trained  and  educated  to  apply  aviation  expertise  to  assess, 

train/educate,  advise,  and  assist  foreign  personnel  in  the  development  and 

application  of  their  aviation  resources  to  meet  their  national  needs,  in  support  of 
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U.S.  interests”  (Department  of  the  Air  Force,  2012b:  10).  While  advising  is  not  a 
new  activity  to  Airmen,  it  has  historically  been  employed  during  conflicts  by 
combat  aviation  advisors  from  within  the  Special  Operations  community, 
particularly  in  IW  and  in  counter  insurgency  (COIN)  operations.  However,  the 
knowledge  and  experience  gained  was  often  lost  after  “draw  downs  or 
conversions  after  each  conflict”  (Read,  2007:42).  To  prevent  having  to  relearn 
the  essential  requirements  of  effectively  working  with  PNs,  the  AF  sought  to 
widen  its  pool  of  AAs  by  using  its  general  purpose  force  (GPF). 

As  stated  by  the  SECAF  and  CSAF  in  2009  The  21st  Century  Air  Force  Irregular 
Warfare  Strategy,  “The  Air  Force  will  establish  a  pennanent  GPF  advisory  capability  for 
steady-state,  protracted  IW  requirements  to  complement  existing  SOF  advisory 
capabilities”  (Department  of  the  Air  Force,  2009: 10).  In  order  to  develop  a  trained  corps 
of  AAs,  the  CSAF  in  2007  directed  the  Air  Education  and  Training  Command  (AETC)  to 
formally  establish  a  training  venue  for  AAs  in  order  for  them  to  acquire  the  necessary 
individual,  team,  and  cultural  and  language  skills  prior  to  deployment.  As  a  result,  the 
Air  Advisor  Academy  was  founded  at  Joint  Base  McGuire-Dix-Lakehurst,  New  Jersey. 
Furthennore,  in  2009,  AETC  was  appointed  as  the  Core  Function  Lead  Integrator 
responsible  for  developing  current,  relevant  instruction  for  those  Airmen  selected  to 
fulfill  AA  roles. 

As  highlighted  in  the  AFGPS  (Figure  1): 

The  USAF  continues  to  develop  a  GPF  Air  Advisor  program  to  help  build  global 
air,  space,  and  cyberspace  partnerships  in  support  of  combatant  commanders’  SC 
and  irregular  warfare  activities.  GPF  Air  Advisors  will  be  prepared  both  to 
operate  independently  and  in  concert  with  other  Services  (in  pennissive 
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environments)  and  to  augment  AFSOF  CAA  and  other  SOF  forces  (in  uncertain 
and  hostile  environments)  (Department  of  the  Air  Force,  2011 :38). 


Figure  1 .  Partner  Nation  Air  Mobility  System  Development 


Source:  Air  Mobility  System  Building  Partnerships  Concept  of  Employment ,  20 10a 


Importance  of  Mobility  Airmen  to  BP/BPC 

“The  BP  mission  of  AMC  is  to  conduct  operations  that  will  train,  advise, 
and  assist  PNs  in  the  development  of  an  Air  Mobility  System  (AMS).  ” 

-  Air  Mobility  Systems  Building  Partnerships  Concept  of  Employment,  2010a:  15. 

As  the  worldwide  leader  in  air  mobility  capability,  Air  Mobility  Command 

(AMC)  answered  this  charge  in  December  2010  via  the  AMC  Programming  Plan  10-01, 

AMC  Building  Partner  Capacity  Unit  Activations  for  Contingency  Response  Wings  at 
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Travis  AFB  and  Joint  Base  McGuire-Dix-Lakehurst  (JB  MDL)  (Air  Mobility  Command, 
2010b).  This  document  directed  the  rapid  creation  of  two  Mobility  Support  Advisory 
Squadrons  (MS  AS)  and  aligned  one  under  each  of  two  U.S.  based  Contingency  Response 
Wings  (CRW)— 615  CRW  at  Travis  AFB,  CA  and  621  CRW  at  JB  MDL.  The  MSAS 
were  assigned  commanders  in  April  2011  which  marked  their  official  implementation. 

As  designed,  each  of  these  squadrons  were  assigned  over  70  personnel  from  nearly  25 
Air  Force  specialties  to  provide  the  requisite  functions  to  establish,  sustain,  or  expand  a 
PN’s  aviation  enterprise  and  infrastructure  (Air  Mobility  Command,  2011c).  Core 
functions,  under  the  umbrella  of  Agile  Combat  Support,  include  mobility  operations,  air 
field  operations,  logistics,  fuels,  vehicle  maintenance,  air  transportation,  supply,  civil 
engineering,  electrical  power  production,  communication,  and  security  forces.  These 
selected  skill  sets  were  placed  into  two  teams  (unit  type  codes)  to  support  a  capabilities- 
based  approach  which  allows  paring  and  tailoring  of  team  size  in  order  to  employ  the 
right-sized  capability  required  to  accomplish  the  BPC  mission. 

Concept  of  MSAS  Employment 

The  MSAS  is  assigned  specific  areas  in  which  to  operate.  Each  squadron  serves 
as  AAs  in  order  to  further  the  aviation  enterprise  development  efforts  of  PN’s.  The  818th 
MSAS,  based  at  JB  MDL,  NJ,  is  aligned  to  work  with  U.S.  Africa  Command 
(USAFRICOM)  and  its  air  component,  3rd  Air  Force  (AFAFRICOM),  and  the  571st 
MSAS,  based  at  Travis  AFB,  CA,  is  aligned  with  U.S.  Southern  Command 
(USSOUTHCOM)  and  its  air  component  12th  Air  Force  (AFSOUTH).  The  MSAS  AAs 
are  charged  with  preparing  for  and  conducting  five  basic  activities  as  required: 
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Assess 


• 

•  Train 

•  Advise 

•  Assist 

•  Equip 

Counter  to  direct  military  action  that  occurs  in  response  to  a  conflict  or  crisis,  the 
approach  used  by  the  MSAS  AAs  consists  of  indirect  action  and  international  relationship 
building  that  is  focused  on  preventing  the  likelihood  of  military  conflict  (Figure  2). 
Particularly,  the  AAs  target  their  PN  engagements  to  take  place  during  Phase  0  -  Shape 
operations  (Figure  3).  This  particular  phase  seeks  to  prevent  a  crisis  by  dissuading 
potential  adversaries  and  to  assure  or  solidify  relationships  with  friends  or  allies,  thereby 
preempting  military  intervention  (Joint  Publication  5-0,  2011b). 


Figure  2.  Irregular  Warfare  Approaches 


Source:  Livingston,  2011 
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Figure  3.  Phasing  Model 
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Source:  Air  Force  Doctrine  Document  2-3,  Irregular  Warfare,  2007 


Joint  Publication  3-0,  Joint  Operations,  further  explains  that  Shape  activities: 

. .  .are  executed  continuously  with  the  intent  to  enhance  international  legitimacy 
and  gain  multinational  cooperation  by  shaping  perceptions  and  influencing 
adversaries’  and  allies’  behavior;  developing  allied  and  friendly  military 
capabilities  for  self-defense  and  multinational  operations;  improving  infonnation 
exchange  and  intelligence  sharing;  providing  U.S.  forces  with  peacetime  and 
contingency  access;  and  mitigating  conditions  that  could  lead  to  a  crisis  (Joint 
Publication  3-0,  201  la:V-8). 


Tasking  Process 

The  essential  link  to  realizing  the  potential  gains  AAs  can  produce  is  the  tasking 
process.  As  outlined  by  AMC  in  the  2010,  Air  Mobility  Systems  Building  Partnerships 
Concept  of  Employment  (AMS  CONEMP),  requests  for  BP  activities  can  be  initiated  in 
several  ways.  Whether  from  a  PN  through  their  associated  U.S.  Embassy  or  from 
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Headquarters  (HQ),  U.S.  Air  Force,  the  Geographic  Combatant  Commander  (GCC)  must 
formally  validate  and  request  an  AA  tasking. 

Prior  to  the  request,  the  GCC  must  coordinate  with  a  variety  of  agencies  such  as 
the  Department  of  State,  U.S.  Embassy  in  the  PN,  other  GCCs,  Secretary  of  the  Air 
Force/International  Affairs  office,  and  Service  Components  to  verify  if  the  request  is  in 
line  with  the  over-arching  U.S.  strategy,  theater,  and  regional  objectives  (AMS 
CONEMP,  2010).  If  it  is,  it  is  validated  and  sent  to  the  Service  Component  to  fill  via  its 
assigned  CRW  or  Contingency  Response  Group  (CRG). 

However,  if  the  Service  Component  is  unable  to  fulfill  the  request,  the  GCC  will 
route  its  request,  via  a  request  for  forces  (RFF),  to  U.S.  Transportation  Command 
(USTRANSCOM).  TRANSCOM  will  review  and  validate  the  request  and  forward  to 
AMC  for  coordinating.  AMC  in  turn  will  task  the  appropriate  CRW,  either  615  CRW  or 
621  CRW,  to  accomplish  the  tasking  via  their  MS  AS.  (Note:  AMC  will  have  only  one 
CRW,  the  621  CRW,  as  of  29  May  2012.)  Should  the  selected  CRW  be  unable  to  fulfill 
the  tasking,  AMC  can  task  another  BP-capable  force  from  outside  the  CRW  or  seek 
assistance  from  the  Air  National  Guard  or  Air  Force  Reserve  (AMS  CONEMP,  2010). 

Once  tasked,  the  MSAS  will  work  with  their  Service  Component  and  the  GCC  to 
determine  the  type  of  support  and  timeline  required.  Through  continuous  communication 
and  coordination,  the  MSAS  will  assemble  the  appropriate  sized  team  to  deploy  forward 
and  initially  assess  the  PN’s  capability  and  capacity.  Based  off  their  findings,  the  MSAS 
will  further  develop  an  engagement  plan  to  achieve  the  GCC’s  desired  outcome  and 
coordinate  further  access  to  the  PN  through  the  GCC  and  appropriate  Embassy  in  order  to 
put  the  plan  to  work. 
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Through  each  visit,  the  MSAS  seeks  to  establish  and  cultivate  a  mutually 
beneficial,  lasting  relationship  with  each  of  their  PN  military  counterparts  while 
simultaneously  seeking  to  contribute  to  the  accomplishment  of  the  GCC’s  Theater 
Security  Cooperation  Plans. 

Anticipated  Benefits  of  BP/BPC 

By  seeking  to  positively  influence  activities  in  this  phase,  preventative  programs 
are  more  efficient  and  effective  than  reactive  efforts  because  they  achieve  the  same  goals 
with  fewer  assets  and  less  commitment  (Teichert,  2009).  Additionally,  by  creating 
programs  to  train  PNs  to  establish  and  maintain  their  own  security,  it  reduces  U.S. 
commitment  of  time,  effort,  money,  and  resources.  Supporting  security  cooperation  and 
foreign  internal  defense  programs  by  assisting,  advising,  training,  and  equipping  can 
provide  capabilities  early  in  a  conflict  when  the  threat  is  smaller,  which  can  serve  to 
strengthen  legitimate  governments  and  spread  democratic  ideals  (Teichert,  2009). 

Furthermore,  a  RAND  Corporation  study  highlighted  that  “many  precautionary 
interventions  can  be  carried  out  for  the  price  of  a  single  remedial  one”  (Vick  et  ah, 
2006:72).  According  to  Teichert  (2009),  the  RAND  study  further  claimed  that  the  cost 
savings  derived  from  preventing  problems  instead  of  intervening  after  problems  have 
expanded  is  so  substantial  that  these  programs  are  worth  pursuing  even  if  they  prevent  an 
American  intervention  in  one  single  major  conflict.  If  preventative  efforts  fail,  and  U.S. 
military  action  is  required,  benefits  from  PN  relationships  like  country  access  and  overfly 
rights  may  provide  necessary  capability  to  intervene  and  limit  overall  time,  effort,  and 
resources  expended. 
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Challenges 

While  the  potential  benefits  and  capabilities  AAs  can  bring  to  bear  are  numerous, 
there  are  inherent  challenges  the  MSAS  face  as  they  continue  to  mature.  They  include: 

•  Lacking  a  universally  recognized  and  utilized  lexicon.  In  reference  to  working 
with  PNs,  common,  interchangeably  used  terms  include:  Security  Cooperation, 
Foreign  Internal  Defense,  Security  Assistance,  Security  Force  Assistance, 

Building  Partnership,  Building  Partnership  Capacity,  etc. 

•  Making  Department  of  State,  Embassy  Country  Teams,  and  Geographic 
Combatant  Commanders  and  their  component  staffs  aware  of  the  MSAS  mission. 

•  Managing  expectations;  relationship  building  with  PNs  takes  time. 

•  Developing  a  “whole  of  government”  approach— coordinating  between  U.S. 
government  entities  to  ensure  organized,  deliberate  efforts  are  made. 

•  Pending  budget  cuts. 

•  Lacking  an  AF  doctrine  document  for  BP/BPC;  an  initial  document  was 
scheduled  and  initially  drafted  but  was  abandoned. 

•  Developing  processes  as  they  go  since  there  is  not  a  GPF  equivalent  to  the  MSAS. 

•  Neither  measuring  criteria  nor  a  reporting  mechanism  has  been  developed  for 
MSAS  to  convey  their  capabilities  and  progress  with  PNs. 

While  current  guidance,  as  highlighted  above,  has  stressed  the  urgency  and  long¬ 
term  benefits  for  undertaking  BP/BPC,  an  ongoing  challenge  inherent  with  building 
relationships  is  measurement.  Establishing,  building,  and  fostering  relationships  with 
PNs  are  time-consuming  endeavors  and  lengthy  in  nature.  Furthermore,  quantifying  and 
conveying  desired  outcomes  of  qualitative  elements  like  trust,  good-will,  and  positive 
feelings  towards  the  U.S.  is  challenging,  which  makes  measurement  of  outcome  very 
subjective.  The  next  chapter  will  outline  the  methodology  this  study  used  to  help  bridge 
the  existing  gap  between  measuring  and  reporting  BPC  activities. 
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III.  Methodology 


In  addition  to  literature  reviews,  this  study  employed  other  research  tools  such  as 
interviews  and  personal  communication  via  voice  and  computer  medium.  The 
infonnation  obtained  from  these  methods  was  used  to  develop  the  foundation  of  the 
study.  Additionally,  to  get  to  the  crux  of  the  issue,  a  Delphi  study  was  conducted 
utilizing  a  pool  of  military  and  non-military  subject  matter  experts  to  serve  as  the  study 
panel.  The  method  used  a  series  of  e-mailed  questionnaires  as  the  tool  for  the  panel  to 
exchange  ideas  and  identify  an  initial  set  of  performance  objective  criteria  the  MS  AS 
could  use  to  measure  their  efforts  and  report  their  performance  towards  meeting  BPC 
goals.  Because  there  is  no  higher  headquarters  sanctioned  method  and  mechanism  yet, 
and  the  delivery  date  is  unknown,  measurement  criteria  are  vital  to  ensure  data  and 
feedback  obtained  during  mission  preparation  and  execution  are  not  lost. 

The  Delphi  Method 

The  Delphi  method  originated  from  research  conducted  by  the  RAND 
Corporation  in  the  1950s.  While  it  was  initially  developed  as  a  decision  making  tool  for 
scientific  and  technological  forecasting  (Sackman,  1974),  the  method  is  used  in  many 
different  environments  to  obtain  the  opinions  of  experts.  It  is  a  “widely  used  and 
accepted  method  for  achieving  convergence  of  opinion  concerning  real-world  knowledge 
solicited  from  experts  within  certain  topic  areas  (Hsu  and  Sandford,  2007: 1).  It  is  a 
flexible  and  iterative  group  communication  process  which  is  used  to  collect  and  distill  the 
anonymous  judgments  of  experts  using  a  series  of  questionnaires  interspersed  with 
feedback  (Skulmoski  et  ah,  2007).  The  questionnaires  are  designed  to  clarify  problems, 
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identify  opportunities  and  solutions,  or  develop  forecasts  (Skulmoski  et  ak,  2007).  The 
process  continues  until  consensus  of  opinion  is  developed  concerning  a  specific  topic 
(Hsu  and  Sandford,  2007). 

According  to  Linstone  and  Turnoff  (2002:3),  the  Delphi  Study  is  “a  method  for 
structuring  a  group  communication  process  so  that  the  process  is  effective  in  allowing  a 
group  of  individuals,  as  a  whole,  to  deal  with  a  complex  problem.”  Stated  differently  by 
one  of  the  originators  of  this  method,  Delphi  is  based  on  the  adage  that  ‘“two  heads  are 
better  than  one,’  or  more  generally,  ‘n-  heads  are  better  than  one’”  (Dalkey,  1972:6). 

Based  on  the  literature  reviewed,  the  Delphi  process  can  be  continuously  iterated 
until  consensus  is  reached.  While  there  is  no  standard  indicator  for  consensus  among 
panelists,  Schiebe  et  al.  (2002:271)  identify  that  in  most  Delphi  studies  “consensus  is 
assumed  to  have  been  achieved  when  a  certain  percentage  of  the  votes  fall  within  a 
prescribed  range — for  example,  when  the  interquartile  range  is  no  larger  than  2-units  on  a 
10-unit  scale.”  Furthermore,  consensus  is  oftentimes  reached  after  three  iterations  (Hsu 
and  Sandford,  2007). 

As  summarized  by  Nic  Underhill  (2004),  the  basic  way  of  conducting  a  Delphi 
study  is: 

Step  1:  The  facilitator  develops  an  initial  questionnaire  and  distributes  it  to  the 
panel. 

This  step  initiates  the  first  round  and  usually  begins  with  an  open-ended 
questionnaire  that  seeks  to  solicit  specific  information  about  a  content  area  from  the 
Delphi  subjects  (Hsu  and  Sandford,  2007). 

Step  2:  Panelists  independently  generate  answers  for  the  questionnaire  and  return 
it  to  the  facilitator. 
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The  answers  the  panel  provides  are  reviewed  and  categorized  and  serve  as  the 
basis  for  the  second  questionnaire. 

Step  3:  The  facilitator  summarizes  the  responses  to  the  first  questionnaire  and 
develops  a  feedback  report  along  with  the  second  set  of  questionnaires 
for  the  panelists. 

This  step  initiates  the  second  round.  Based  on  the  inputs  received,  the  facilitator 
categorizes  the  inputs  and  requests  that  the  panel  rank  or  rate  the  importance  of  the 
categories  formed  based  off  of  their  initial  answers.  Furthermore,  the  facilitator  asks  the 
panel  members  to  review  their  round  one  responses  to  ensure  their  opinions  were 
properly  captured,  to  adjust  their  answers  as  required,  and  to  provide  additional  feedback. 

Step  4:  Referencing  the  feedback  report,  panelists  independently  evaluate  earlier 
responses  and  independently  vote  on  the  second  questionnaire. 

Panelists  are  given  initial  insight  into  answers  provided  by  the  entire  panel  and 
given  the  opportunity  to  refine,  retract,  or  expand  their  previous  answers.  Additionally, 
they  may  be  asked  to  “rate  or  rank-order  items  based  off  the  responses  to  the  first 
questionnaire  in  order  to  establish  preliminary  priorities  among  items”  (Hsu  and 
Sandford,  2007:2-3). 

Step  5:  The  facilitator  summarizes  the  responses  to  the  second  questionnaire  and 
develops  feedback  report  along  with  the  third  set  of  questionnaires  for 
the  panelists. 

This  step  initiates  the  third  round.  Similar  to  round  two,  the  facilitator  categorizes 
the  inputs  and  requests  the  panel  to  rank  or  rate  the  importance  of  the  categories  formed 
based  on  the  round  two  answers.  Furthennore,  the  facilitator  asks  the  panel  members  to 
review  the  round  two  responses  to  ensure  their  opinions  were  properly  captured,  to  adjust 
their  answers  as  required,  and  to  provide  additional  feedback. 
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Step  6:  Referencing  the  feedback  report,  panelists  independently  evaluate  earlier 
responses  and  independently  vote  on  the  third  questionnaire. 

Panelists  are  provided  the  collective  answers  from  round  two  and  given  the 
opportunity  to  adjust  their  previous  answers.  Also,  panel  members  are  asked  to  comment 
on  the  emerging  collective  perspective.  If  consensus  is  reached,  and,  ultimately,  the 
research  question  is  answered,  the  process  stops. 

Step  7 :  The  facilitator  develops  a  final  summary  report  and  sends  it  to  the 
group  for  a  final  review. 

This  step  initiates  the  fourth  and  final  round.  The  facilitator  analyzes  and 
summarizes  the  third  round  inputs.  Then,  the  information  is  returned  to  the  panel  for  a 
final  review.  This  allows  for  thanking  the  panelists  for  their  participation  and  allows  for 
closure. 

While  many  purely  qualitative  studies  use  one  or  more  of  the  most  common 
qualitative  research  methods  such  as  participant  observation,  in-depth  interviews,  and 
focus  groups  (Mack  et  ah,  2005),  the  researcher’s  advisor  recommended  that  the  Delphi 
technique  may  be  better  suited  for  this  study.  This  led  to  a  review  of  a  variety  of 
journals,  research  papers,  and  books  published  on  the  Delphi  method  and  its  use  as 
compared  to  other  traditional  survey  methods.  Due  to  the  flexibility  of  design  and 
execution,  the  Delphi  technique  was  deemed  the  best  tool.  Since  this  method  doesn’t 
require  face-to-face  interaction,  qualified  panel  members  from  a  variety  of  backgrounds 
and  experience  from  around  the  globe  can  participate.  This  fact,  coupled  with  anonymity 
and  equally  weighted  responses,  enhances  the  likelihood  of  honest  and  open  perspectives 
exchanges,  idea  generation,  and  potential  solutions  to  existing  or  potential  questions  or 
issues. 
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There  are,  however,  some  challenges  associated  with  the  Delphi  technique.  It 
often  has  poor  or  declined  response  rates  to  each  questionnaire  and  subsequent  difficulty 
obtaining  robust  feedback  (Hsu  and  Sandford,  2007).  The  quality  of  outcome  is 
dependent  on  the  experiential  knowledge  of  the  panel  (Powell,  2003),  and  it  is  time 
consuming  to  conduct.  Figure  4  shows  strengths  and  limitations  of  a  Delphi. 


Figure  4.  Delphi  Strengths  and  limitations 


^Stren^ths 


Limitations 


•  Rapid  consensus. 

•  Participants  can  live  anywhere  in  the 
world. 

•  Avoids  group-think 

•  Flexible  with  no  set  meeting  times, 
only  an  agreeable  deadline. 

•  Information  can  be  very  powerful  in 
predicting  future  trends  and  events. 

•  Sain  consensus  on  a  large  scale 
(provincial  to  national  to  global). 

•  Cost  effective. 

•  More  detailed  than  typical  survey 
responses. 


•  Quality  of  responses  depend  on  panel. 

•  Researchers  must  choose  participants 
selectively. 

•  Time  consuming.  This  process  is  rarely 
used  by  researchers/agencies. 

•  Panel  may  lose  interest  in  topic  if  con¬ 
sensus  is  not  reached  within  reasonable 
time. 

•  Panel  may  lose  cohesiveness  overtime. 

•  Interactions  between  participants  and 
researcher  are  not  face-to-face,  depending 
on  subject  matter  this  fact  can  hinder  or 
assist  in  information  gathering. 


Source:  Skagfeld  and  Derbyshire,  n.d. 


By  using  the  opinions  of  experts  who  offer  a  mixture  of  relevant  perspectives  and 
experiences,  the  panel’s  collective  voice  can  help  to  identify  plausible  solutions  to  the 
research  questions:  1)  What  minimum  criteria  would  you  recommend  the  MS  AS  use  to 
measure  their  effectiveness  at  accomplishing  their  Building  Partnership  Capacity 
mission?  and  2)  What  reporting  mechanism  would  you  recommend  the  MSAS  use  to 
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track  and  report  their  unit  effectiveness?  The  next  chapter  will  describe  the  application  of 
Delphi  methodology  and  highlight  the  results  obtained  from  the  study. 
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IV.  Analysis  and  Results 


Application  of  Methodology 

Prior  to  accomplishing  Step  1,  the  issue  was  properly  formulated  and  framed. 
There  were  a  variety  of  issues  that  were  being  simultaneously  worked  by  each  MS  AS. 
These  included  obtaining  facilities,  personnel,  funding  streams,  developing  an 
organizational  structure,  tasking  process,  and  advertising  their  existence  and  capabilities. 
However,  the  researcher’s  focus  narrowed  after  receiving  the  following  input  from  the 
818  MSAS  director  of  operations: 

We  could  use  some  help  on  how  to  measure  success  in  the  short  and  midterm  in 
the  BP/BPC  mission— a  mission  that  is  inherently  long-tenn.  That's  going  to  be  a 
big  challenge  for  us  as  we  try  to  show  our  capabilities  to  an  Air  Force  where 
everything  revolves  around  metrics  and  impact  (P.  Smith,  personal 
communication,  July  28,  2011). 

So,  after  considering  the  sum  of  the  issues,  a  research  plan  was  devised  on  how  to 
best  answer  that  question.  Then,  dialogue  continued  with  the  MSAS.  This  led  to  the 
exchange  of  names  and  offices  of  other  subject  matter  experts  at  Headquarters  Air  Force 
(HAF),  Air  Mobility  Command  (AMC),  Air  Force  Special  Operations  Command 
AFSOC),  and  Air  Education  and  Training  Command  (AETC)  who  were  in-tum  contacted 
and  consulted.  In  addition  to  the  information  garnered  from  these  contacts,  the  issue  was 
further  refined  by  conducting  a  more  in-depth  literature  review. 

Due  to  a  lack  of  established  standards  the  MSAS  could  reference,  it  was  necessary 
to  find  appropriate  criteria  to  constitute,  measure,  and  report  success.  Since  the  MSAS 
mission  is  largely  relationship-oriented,  a  variety  of  reports  and  related  sources  of  data 
from  organizations  such  as  USAID,  the  Peace  Corps,  and  various  non-governmental 
organizations  that  work  to  form  relationships  within  communities  both  within  the  U.S. 
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and  abroad  were  referenced.  From  the  data  obtained,  it  was  confirmed  that  observer 
subjectivity  contributes  to  a  long-standing  challenge  with  finding  ways  to  quantify  or 
report  the  perceived  effectiveness  in  partner  engagements.  This  substantiated  the 
inherent  difficulty  that  extends  to  the  MSAS  as  well.  Furthennore,  contact  was  made 
with  persons  from  within  AFSOC  and  former  members  of  the  6th  Special  Operations 
Squadron,  the  most  experienced  unit  of  aviation  advisors  within  the  AF,  regarding  how 
the  unit  tracks,  measures,  and  reports  their  PN  engagements.  Their  inputs  verified  that 
formalized  measurement  criteria  are  non-existent.  The  unit  plans  and  executes  its  PN 
engagements  with  pre-determined  outcomes  in  mind  as  dictated  by  their  tasking 
authority.  Following  the  completion  of  their  PN  engagement,  the  primary  tool  used  to 
report  unit  actions,  efforts,  and  outcomes  are  after  action  reports.  As  a  result  of  the 
variety  of  information  obtained,  the  primary  research  questions  were  developed,  which 
served  as  the  basis  for  the  Delphi  study.  With  this  foundation  in  place,  the  next,  perhaps 
most  essential  action  was  to  assemble  a  panel  of  experts. 

Selection  of  Experts  for  the  Panel 

Skulmoski  et  ah,  (2007:3)  state  that  “selecting  research  participants  is  a  critical 
component  of  Delphi  research  since  it  is  their  expert  opinions  upon  which  the  output  of 
the  Delphi  is  based.”  Hsu  and  Sandford  (2007:3)  contend  that  “choosing  the  appropriate 
subjects  is  the  most  important  step  in  the  entire  process  because  it  directly  relates  to  the 
quality  of  the  results  generated.”  Furthermore,  Gordon  (1994:6)  claims  that  the  “key  to  a 
successful  Delphi  study  lies  in  the  selection  of  participants.”  While  proper  panel 
selection  is  essential  to  the  Delphi  study,  literature  review  has  identified  that  there  is  no 
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standard  list  of  criteria  to  use  when  selecting  Delphi  subjects.  After  reviewing  several 
sources,  Skulmoski  et  al.’s  (2007)  criteria  were  used.  Their  view  purports  participants 
should  meet  the  following  four  requirements  to  be  considered  “experts”:  1)  knowledge 
and  experience  with  the  issues  under  investigation,  2)  capacity  and  willingness  to 
participate,  3)  sufficient  time  to  participate,  and  4)  effective  communication  skills 
(Skulmoski  et  al.,  2007).  Additionally,  Scheele  proposes  that  there  are  three  types  of 
panelists  that  help  create  a  successful  mix  of  panelists.  These  are  “stakeholders,  those 
who  are  or  will  be  directly  affected;  experts,  those  who  have  an  applicable  specialty  or 
relevant  experience;  and  facilitators,  those  who  have  skills  in  clarifying,  organizing, 
synthesizing,  stimulating”  (Linstone  and  Turoff,  2002:65). 

Based  on  the  above  criteria,  the  researcher  targeted  persons  in  key  positions  with 
BPC  and/or  AA  experience  from  HAF,  AMC,  AETC,  AFSOC,  and  the  CRWs.  For  a 
combat  support  perspective,  experts  included  two  currently  deployed  Air  Force  A  As 
working  with  the  Afghan  National  Air  Force  and  an  Ainnan  that  served  on  two  365-day 
Joint  Expeditionary  Taskings  working  with  the  Afghan  National  Army  and  Police.  Other 
panelists  included  a  recently  retired  State  Department  diplomat  that  served  a  year  in  Iraq 
as  an  advisor  on  a  Provincial  Reconstruction  Team  and  a  School  of  Advanced  Air  and 
Space  Studies  graduate  who  recently  published  literature  on  this  topic.  A  total  of  20 
individuals  were  selected  to  participate  in  the  study,  and  all  accepted. 
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Delphi  Study  Execution 

Step  1 

With  the  Delphi  panel  in  place,  the  initial  questionnaire  was  drafted.  The  research 
questions  were  developed  by  the  researcher  in  consultation  with  BP/BPC  subject  matter 
experts  and  Air  Force  Institute  of  Technology  (AFIT)  faculty  research  advisor.  As 
recommended  by  Linstone  and  Turoff  (2002:89),  in  order  to  eliminate  potential 
confusion,  “each  questionnaire  should  be  pretested  on  coworkers  who  have  not  been 
involved  in  the  design.”  The  final  draft  of  the  questionnaire  was  pretested  by  the 
researcher’s  classmates,  AFIT  research  advisor,  and  select  experts  and  was  deemed  clear, 
concise,  and  effective  towards  obtaining  valuable  panel  responses. 

The  questionnaire  (Appendix  A)  along  with  an  MSAS  tri-fold  depicting  the  unit 
and  its  mission  (Appendix  B)  was  personally  sent  via  e-mail  to  each  of  the  20  panel 
members.  While  there  were  other  distribution  means  available,  current  literature 
contends  that  e-mail  delivery  is  highly  effective.  Per  Sheehan  and  McMillan  (1999),  e- 
mail  tends  to:  1)  have  higher  response  rates  over  post  mail,  2)  promotes  faster  response 
times,  and  3)  respondents  seem  more  willing  to  reply  to  open-ended  questions.  This 
initiated  Round  1 . 

Step  2 

Ten  duty  days  were  allotted  for  the  Round  1  questionnaire  to  be  completed  by  the 
panel.  Of  the  questionnaires  distributed,  19  of  20  were  returned  within  the  timeframe  set 
and  resulted  in  a  95%  response  rate.  The  last  questionnaire  was  returned  after  the  cut-off 
and  not  included  in  the  first  round  results. 
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Step  3 


From  the  responses  received,  a  content  analysis  was  conducted  whereby  the 
panel’s  demographic  detail  was  analyzed  (Table  1).  Of  the  20  panel  members,  four 
served  as  combat  aviation  advisors  with  the  6th  Special  Operations  Squadron  (6  SOS), 
three  served  as  AAs  in  Iraq  and  Afghanistan,  four  are  currently  serving  as  AAs  (two  in 
Afghanistan,  one  for  Africa,  and  one  for  South  America),  two  served  as  advisors  to 
ground  forces  in  Iraq  and  Afghanistan,  one  served  as  an  Air  Attache,  and  one  served  as  a 
Security  Assistance  Officer.  The  remaining  live  panelists  serve  as  strategic  policy 
developers  or  are  in  education  and  training  roles.  Of  note,  one  of  the  educators  is  a 
retired  political/military  affairs  strategist  who  also  directs  the  AF  Special  Operations 
School’s  Building  Partner  Aviation  Capacity  Seminar. 


Table  1,  Panel’s  Demographic  Detail 


Position 

Number 

Combat  Aviation  Advisor 

4 

Air  Advisor  (AFG  &  IRQ) 

7 

Ground  Advisor  (AFG  &  IRQ) 

2 

Air  Attache 

1 

Security  Assistance  Officer 

1 

Policy,  Education  &  Training 

5 

Total 

20 

Collectively,  the  panelists  who  have  served  in  an  advisory  capacity,  to  include  the 
Air  Attache  and  Security  Assistance  Officer,  have  amassed  400  months,  or  over  33  years, 
of  advising  time  to  PNs.  Due  to  the  varying  and  valuable  perspectives  each  of  the  panel 
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members  offers,  the  content  analysis  team  detennined  that  all  respondents’  inputs  would 
receive  the  same  weight  throughout  the  study. 

From  the  inputs  received,  a  content  analysis  was  conducted  and  a  master  list  of  66 
criteria  was  developed.  The  group’s  responses  were  categorized  into  seven  main  subject 
areas  containing  44  criteria  for  the  first  research  question  and  six  main  subject  areas 
containing  22  criteria  for  the  second  research  question. 

The  results  from  the  first  questionnaire  were  summarized  and  presented  to  the 
panel  to  provide  feedback  and  set  the  stage  for  the  next  round.  The  criteria  were  arranged 
from  highest  percentage  of  mentions  to  lowest  percentage  (Table  2).  This  was  the  basis 
for  the  second  Delphi  questionnaire. 
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Table  2.  Categorized  Responses  and  Frequency 


Original  question:  What  minimum  criteria  (i.e.  performance  objectives,  key  indicators,  etc.) 
would  you  recommend  the  MSAS  use  to  measure  their  effectiveness  at  accomplishing  their 
Building  Partnership  Capacity  mission? 


Category 

Frequency 

Ability  to  gain/maintain  access  to  Partner  Nation  (PN) 

14 

Subcategory: 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to 
information,  facilities,  airfields,  equipment,  etc.) 

9 

Ability  to  establish  key  contacts 

4 

Ability  to  make  repetitive/persistent  visits  with  PN 

1 

Actions  tied  to  U.S.  National/Strategic  policy  goals 

8 

Subcategory: 

MSAS  engagements  are  in  support  of  U.S.  National  Security 
policy,  Geographic  Combatant  Commander  (GCC)  and 

Component  Plans  (e.g.  Theater  Security  Cooperation  Plans, 
Campaign  Plans,  and  Air  Force  Component  Campaign  Support 

Plan)  and  Ambassador  Country  Plans  (e.g.  Mission  Strategic  and 
Resource  Plan) 

6 

MSAS  actions  are  integrated  with  other  U.S.  Government  agency 
actions 

2 

Post-engagement  outcomes 

6 

Subcategory: 

MSAS  contributed  towards  PN's  increased  capability  and  capacity 
(e.g.  response  during  crisis,  stable  or  increasing  mission  capability 
rate,  sustained  personnel  training/task  proficiency, 
interoperability  with  U.S.)  as  reported  by  country  team,  GCC  staff 

3 

Extent  to  which  MSAS  accomplished  policy  objectives  (e.g.  GCC 
end  states  met) 

2 

Notable  improvement/development  of  key  relationships  with  PN 
(e.g.  within  government,  military) 

1 

Ability  to  further  develop  MSAS  capability 

6 

Subcategory: 

Ability  to  identify  and  recruit  personnel  for  MSAS  (e.g.  locate, 
train,  develop,  track/place)  and  seek  to  establish  a  formal  career 
track  and  career  progression/promotion  opportunities 

2 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g. 
language/culture  familiarization,  general  advising  skills,  etc.) 

2 

Ability  to  develop  internal  processes  that  support  MSAS  mission 
(e.g.  record  keeping  for  information  continuity) 

1 

Ability  to  develop  key  relationships  (e.g.  other  military  services, 
Department  of  State,  interagency  organizations,  regional  centers 
like  the  Marshall  Center  in  Europe) 

1 

Ability  to  set  realistic  goals  for  PN  and  provide  the  tools/skills  to  attain  them 

4 

Subcategory: 

Ability  to  develop  plans,  milestones,  projects  the  PN  can 
accomplish  within  realistic  timeframe 

4 
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Table  2.  Categorized  Responses  and  Frequency  (cont). 


Category 

Frequency 

Ability  to  develop  standards  of  performance 

4 

Subcategory: 

Ability  to  develop  achievable  Mission  Essential  Task  Lists  and 
performance  standards  for  PN 

4 

Number  of  taskings  received  vs.  number  of  taskings  completed 

2 

Subcategory: 

Requestor  and  types  of  requests  received  (e.g.  Assess,  Train, 

Advise,  Assist,  Equip) 

1 

Frequency  and  duration/longevity  interactions  with  PN 

1 

Total 

44 

Original  question:  What  reporting  mechanism  would  you  recommend  the  MSAS  use  to  track 
and  report  their  unit  effectiveness? 


Category 

Frequency 

After  Action  Reports 

5 

Centralized  data  collection  and  dissemination  systems 

5 

Subcategory: 

Make  entries  into  GCC/Air  Force  Theater  Security  Cooperation 
Management  Information  System  (TSCMIS) 

4 

Make  entries  into  Joint  Lessons  Learned  Information  System 
(JLLIS) 

1 

Spreadsheets 

5 

Subcategory: 

Track  PN  progress  following  MSAS  engagements  using  a  numeric 
rating  scale  (e.g.  1  -  5  scale:  (0)  -  No  capability  and  (5)  -  Can 
complete  without  assistance 

5 

Feedback  from  Surveys 

3 

Subcategory: 

Survey  results  from  GCC,  Component  staff,  Country  team,  PN 

3 

Formal  reports 

3 

Subcategory: 

Submit  annual/semi-annual  reports  to  GCC,  Component  staff,  and 
Air  Mobility  Command  through  MSAS  chain  of  command 

3 

Unit  Readiness  reporting 

1 

Subcategory: 

Report  unit  mission  readiness  monthly  through  Status  of 

Resources  and  Training  System  (SORTS) 

1 

Total 

22 

In  addition,  the  panel  was  offered  the  opportunity  to  rate  each  of  the  criterion 
from  the  first  questionnaire  by  level  of  importance  utilizing  a  five-point  Likert  scale, 
where  5=Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance, 
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l=Unimportant  (Figure  5).  This  technique  was  utilized  because  other  researchers  found 
that  the  rating-scale  method  is  “quick,  easy  to  comprehend,  and  psychologically 
comforting”  for  participants  to  perform  and,  therefore,  more  likely  to  result  in  returned 
questionnaires  (Schiebe  et  ah,  2002:267). 


Figure  5.  Delphi  Questionnaire  2 

1 )  Please  review  each  of  the  following  items  identified  in  Questionnaire  #  1  as  minimum  criteria  the 
MS  AS  could  use  to  measure  their  effectiveness  at  accomplishing  their  Building  Partnership  Capacity 
mission.  If  you  wish  to  add  comments  expressing  agreement,  disagreement,  or  clarification  concerning 
an  item,  please  do  so  in  the  space  provided.  Brevity  and  clarity  will  facilitate  analysis. 

2)  Please  rate  the  following  items  by  level  of  importance  as  you  perceive  them  at  this  time. 


5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little 

mportance,  l=Unimportant 

Criteria 

Rating 

Comments 

Ability  to  assess  Partner  Nation  (PN)  capability/capacity  (e.g.  access  to 
information,  facilities,  airfields,  equipment,  etc.) 

Ability  to  establish  key  contacts  within  PN  government,  military,  etc. 

Ability  to  make  repetitive/persistent  visits  with  PN 

After  conducting  a  review  and  pretest  by  classmates  and  AFIT  research  advisor, 
the  questionnaire  was  distributed  to  the  entire  panel  of  experts  (Appendix  C).  This 
initiated  Round  2. 


Step  Four 

Ten  duty  days  were  allotted  for  the  Round  2  questionnaire  to  be  completed  by  the 
panel.  Of  the  questionnaires  distributed,  20  of  20  were  returned  within  the  timeframe  set, 
resulting  in  a  100%  response  rate. 
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Step  Five 

According  to  Hsu  and  Sandford  (2007:4),  “the  major  statistics  used  in  Delphi 
studies  are  measures  of  central  tendency  (means,  median,  and  mode)  and  level  of 
dispersion  (standard  deviation  and  inter-quartile  range)  in  order  to  present  infonnation 
concerning  the  collective  judgments  of  respondents.”  For  studies  using  Likert-scales, 
they  recommend  the  use  of  median  score  (Hsu  and  Sandford,  2007).  Gordon  (n.d.) 
advises  that  the  group  judgment  should  be  based  on  the  median  since  it  is  less  influenced 
by  extreme  answers.  He  also  states  that  the  spread  of  opinion  should  be  represented  by 
showing  the  range  of  responses,  often  the  interquartile  range  since  it  is  the  range  that 
contains  the  answers  of  the  middle  50%  of  the  respondents  (Gordon,  n.d.). 

Since  the  Delphi  method  seeks  to  gain  group  consensus,  an  indicator  must  be 
used.  As  Keeney  et  al.  (2006)  noted,  the  Delphi  literature  does  not  provide  a 
standardized  way  of  selecting  measures  of  consensus.  However,  Plinske  and  Packard 
(2010)  identified  a  common  guideline  that  an  IQR  of  20%  of  the  rating  scale  is  a 
conservative  but  acceptable  criterion  for  determining  consensus.  Furthermore,  Plinske 
and  Packard  (2010)  recommend  analyzing  responses  for  stability  as  well  as  consensus. 
They  cite  Scheibe  et  al.  (1975)  who  defined  stability  as  a  15%  or  less  change  in  responses 
between  rounds.  For  this  study,  consensus  and  stability  was  detennined  by  following 
these  guidelines.  Particularly,  based  on  the  use  of  a  5-point  Likert  scale,  an  IQR  of  1  or 
less  was  an  indicator  of  consensus. 

Utilizing  this  infonnation,  a  content  analysis  was  conducted  on  the  Round  2 
questionnaire  responses,  and  a  statistical  summary  to  measure  central  tendency  using  the 
median  score  and  level  of  dispersion  via  the  IQR  was  calculated  via  Microsoft  Excel  for 
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each  criterion.  Based  on  the  group’s  ratings,  consensus  was  reached  on  nine  criteria  for 
the  first  research  question  and  four  criteria  for  the  second  research  question  (Table  3). 
The  criteria  were  listed  in  order  based  on  the  arithmetic  median  and  IQR  (the  range  that 
contains  the  answers  of  the  middle  50%  of  the  respondents)  of  the  group’s  responses. 


Table  3.  Criteria  that  Consensus  was  Reached  On  in  Round  2 


What  minimum  criteria  would  you  recommend  the  MSAS  use  to  measure 
their  effectiveness  at  accomplishing  their  Building  Partnership  Capacity 
mission? 

R2  Median 

R2  IQR 

MSAS  contributed  towards  Partner  Nation's  (PN)  increased  capability  and 
capacity  (e.g.  response  during  crisis,  stable  or  increasing  mission  capability 
rate,  sustained  personnel  training/task  proficiency,  interoperability  with 

U.S.)  as  reported  by  country  team,  GCC  staff 

5 

1 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g.  language/culture 
familiarization,  general  advising  skills,  etc) 

5 

1 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to  information,  facilities, 
airfields,  equipment,  etc.) 

5 

1 

MSAS  engagements  are  in  support  of  U.S.  National  Security  policy, 

Geographic  Combatant  Commander  (GCC)  and  Component  Plans  (e.g. 

Theater  Security  Cooperation  Plans,  Campaign  Plans,  and  Air  Force 

Component  Campaign  Support  Plan)  and  Ambassador  Country  Plans  (e.g. 
Mission  Strategic  and  Resource  Plan) 

5 

1 

Ability  to  develop  internal  processes  that  support  MSAS  mission  (e.g.  record 
keeping  for  information  continuity) 

4 

0.25 

Ability  to  develop  achievable  Mission  Essential  Task  Lists  and  performance 
standards  for  PN 

4 

1 

Requestor  and  types  of  requests  received  (e.g.  Assess,  Train,  Advise,  Assist, 
Equip) 

4 

1 

Frequency  and  duration/longevity  of  interactions  with  PN 

4 

1 

Ability  to  develop  key  relationships  (e.g.  other  military  services,  Department 
of  State,  interagency  organizations,  regional  centers  like  the  Marshall  Center 
in  Europe) 

4 

1 

What  reporting  mechanism  would  you  recommend  the  MSAS  use  to  track 
and  report  their  unit  effectiveness? 

R2  Median 

R2  IQR 

After  Action  Reports 

5 

1 

Survey  results  from  GCC,  Component  staff,  Country  team,  PN 

4 

1 

Report  unit  mission  readiness  monthly  through  Status  of  Resources  and 
Training  System  (SORTS) 

3 

0.5 

Make  entries  into  Joint  Lessons  Learned  Information  System  (JLLIS) 

3 

1 
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A  third  Delphi  questionnaire  included  the  results  from  the  second  questionnaire, 
which  were  summarized  and  presented  to  provide  feedback  and  a  snap-shot  of  what  the 
group  was  thinking.  Particularly,  it  contained  individual  Round  2  scores  in  relation  to  the 
group  median  and  IQR  as  well  as  any  pertinent  comments  submitted  by  the  panel.  This 
questionnaire  was  designed  to  ask  each  participant  to  review  the  data,  to  reconsider 
individual  responses  as  compared  to  the  group’s  responses,  and  to  support  their 
respective  position  as  required  for  each  criterion — for  both  items  that  reached  and  did  not 
reach  consensus  (Figures  6  and  7).  After  conducting  a  review  and  pretest  by  the 
researcher’s  classmates  and  AFIT  research  advisor,  the  third  questionnaire  was  approved 
and  distributed  to  the  panel  (Appendix  D). 
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Figure  6.  Delphi  Questionnaire  3a. 


Below  are  the  results  from  the  previous  questionnaire.  Based  on  the  group' s  ratings,  consensus  was 
reached  on  9  criteria  for  the  first  research  question  and  4  criteria  for  the  second  research  question. 

The  criteria  below  were  listed  in  order  based  on  the  arithmetic  median  and  interquartile  range  (die  range  that 
contains  the  answers  of  the  middle  50  percent  of  the  respondents)  of  the  group’s  responses.  For  this 
study,  an  interquartile  range  (IQR)  of  1  or  less  is  an  indicator  of  consensus. 

Task:  First,  review  the  ratings  in  left-hand  column  for  both  sections.  Then,  select  “Yes”  or  “No”  in 
the  box  below.  If  you  select  “Yes,”  then  you  are  finished  with  this  section.  If  you  select  “No,”  please  re¬ 
rank  and  comment  as  needed.  Be  sure  to  use  the  same  5-point  rating  scale  from  the  last  round.  Once 
completed,  continue  on  to  Part  2  -  Questionnaire  3. 

I  agree  with  ratings  as  determined  by  the  group  in  Round  2: 


Yes  (Proceed  to  part  2) 

No  (Re-rank,  comment,  then  proceed  to  Part  2) 

Section  1:  The  minimum  criteria  the  MSAS  could  use  to  measure  their  effectiveness  at  accomplishing 
their  Building  Partnership  Capacity  mission. 


5  =Verv  Important,  4=Important,  3=Moderatelv  Important,  2=Of  Little  Importance,  l=Unimportant 


Your 

Rating 

Group's 

Rating/IQR 

Criteria 

New  Rating/Com ment(s) 

5 

5/1 

MSAS  contributed  towards  Partner  Nation's  (PN)  increased 
capability  and  capacity  (e.g.  response  during  crisis,  stable 
or  increasing  mission  capability  rate,  sustained  personnel 
training/task  proficiency,  interoperability  with  U.S.)  as 
reported  by  country  team,  GCC  staff 

5 

5/1 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g. 
language/culture  familiarization,  general  advising  skills, 
etc) 

5 

5/1 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to 
information,  facilities,  airfields,  equipment,  etc.) 
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Figure  7.  Delphi  Questionnaire  3b. 


The  items  below  did  not  reach  consensus  in  the  last  round.  Therefore,  items  in  section  1  and  2  below 
should  be  reviewed  and  re-assessed  by  the  group  in  an  effort  to  reach  consensus.  The  criteria  were 
analyzed  using  the  arithmetic  median  of  the  responses  given  in  Questionnaire  #2. 

Task:  Please  re-rate,  as  required,  the  items  in  both  sections  by  level  of  importance  as  you  perceive 
them  at  this  time  using  the  5-point  rating  scale  below.  If  you  want  to  keep  your  original  rating,  please 
add  “No  Change”  to  the  comments  section.  Add  brief,  concise  comments  for  clarification. 

Section  1:  The  minimum  criteria  the  MS  AS  could  use  to  measure  their  effectiveness  at  accomplishing 
their  Building  Partnership  Capacity  mission.  There  are  8  items  included. 


5  =Verv  Important,  4=Important,  3=Moderatelv  Important,  2=Of  Little  Importance,  l=Unimportant 

Your 

Rating 

Group's 
Rating 
from 
Round  2 

Criteria 

Group's  Comments 

New  Rating/Comment(s) 

3 

5 

Abilityto  establish  key  contacts 
within  PN  government,  military, 

etc. 

-  Relationships  are#l  priority. 

-  This  does  not  tell  me  how 

effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

-No  substantial  contacts,  no 
mission. 

-  Dept  of  State  responsibility. 
-Ways  (Ends,  Ways,  Means). 

3 

4 

Abilityto  make 

repetitive/persistent  visits  with  PN 

-  Key  to  MSAS  mission. 

-  Means  (Ends,  Ways,  MSACS)- 
-This  does  not  tell  me  how 
effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

-  Establishes/maintains 
relationships  and  results  in 
minimal  re-learning  of 
requi  rem  ents/expectations . 

-This  criterion  depends  entirely 
on  the  nature  of  why  return  visit 
is  warranted. 

-  Depends  on  what  you  want  to 
do  with  the  country. 

Step  Six 

Ten  duty  days  were  allotted  for  the  Round  3  questionnaire  to  be  completed  by  the 


panel.  Of  the  questionnaires  distributed,  20  of  20  were  returned  within  the  timeframe  set, 


resulting  in  a  100%  response  rate. 
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Step  Seven 

While  the  Delphi  technique  is  an  iterative  process  that  seeks  to  develop  group 
consensus,  much  of  the  Delphi  literature  purports  that,  in  most  cases,  three  rounds  are 
often  sufficient  to  collect  the  needed  information  and  obtain  consensus  (Hsu  and 
Sandford,  2007).  Sumsion  (1998)  also  warns  that  after  two  or  three  rounds,  participants 
may  become  fatigued,  and  that,  after  three  rounds,  stability  and  consensus  should  have 
been  reached.  Based  on  this  information,  the  panel’s  high  participation  rate,  the  stability 
of  responses,  and  the  consensus  achievement  on  the  majority  of  criteria,  the  Delphi  study 
was  concluded.  After  analyzing  the  third  questionnaire,  a  final  report  summarizing 
Round  3  findings  was  generated  and  returned  to  each  panel  member  as  a  means  of 
closure  and  to  recognize  their  collective  participation  to  this  study  (Appendix  E). 

As  in  the  previous  rounds,  data  analysis  was  conducted  on  the  responses  the  panel 
provided  in  questionnaire  3.  After  reviewing  the  ratings  and  comments  from  round  2,  the 
group  reevaluated  their  ratings  and  reached  consensus  on  15  of  the  16  criteria  for  the  first 
research  question — an  increase  from  nine  items  in  round  2.  Consensus  was  reached  for 
six  of  the  seven  criteria  for  the  second  research  question — an  increase  from  four  items  in 
round  2.  A  total  of  76  rating  changes  were  made  for  the  first  research  question,  and  38 
rating  changes  were  made  for  the  second  research  question.  Of  the  panel  members,  1 1 
agreed  with  the  collective  group  ratings  from  round  2.  Finally,  the  overall  number  of 
panel  member  comments  provided  was  noticeably  higher  in  round  3  than  in  round  2.  Of 
the  panel,  12  members  made  comments  in  questionnaire  2,  and  18  made  comments  in 
questionnaire  3 — a  50%  increase.  A  summary  of  round  2  and  3  median  and  IQR  scores 
are  provided  in  Table  4. 
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Table  4.  Summary  of  Median  and  IQR  Scores 


What  minimum  criteria  would  you  recommend  the  MSAS 
use  to  measure  their  effectiveness  at  accomplishing  their 
Building  Partnership  Capacity  mission? 

Round  2 

Median 

Round  2 

IQR 

Round  3 

Median 

Round  3 

IQR 

MSAS  contributed  towards  Partner  Nation's  (PN) 
increased  capability  and  capacity  (e.g.  response  during 
crisis,  stable  or  increasing  mission  capability  rate, 
sustained  personnel  training/task  proficiency, 
interoperability  with  U.S.)  as  reported  by  country  team, 

GCC  staff 

5 

1 

5 

0 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g. 
language/culture  familiarization,  general  advising  skills, 
etc) 

5 

1 

5 

1 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to 
information,  facilities,  airfields,  equipment,  etc.) 

5 

1 

5 

0 

MSAS  engagements  are  in  support  of  U.S.  National 

Security  policy,  Geographic  Combatant  Commander  (GCC) 
and  Component  Plans  (e.g.  Theater  Security  Cooperation 
Plans,  Campaign  Plans,  and  Air  Force  Component 

Campaign  Support  Plan)  and  Ambassador  Country  Plans 
(e.g.  Mission  Strategic  and  Resource  Plan) 

5 

1 

5 

0 

Ability  to  develop  internal  processes  that  support  MSAS 
mission  (e.g.  record  keeping  for  information  continuity) 

4 

0.25 

4 

0 

Ability  to  develop  achievable  Mission  Essential  Task  Lists 
and  performance  standards  for  PN 

4 

1 

4 

0.25 

Requestor  and  types  of  requests  received  (e.g.  Assess, 

Train,  Advise,  Assist,  Equip) 

4 

1 

4 

0 

Frequency  and  duration/longevity  of  interactions  with  PN 

4 

1 

4 

0 

Ability  to  develop  key  relationships  (e.g.  other  military 
services,  Department  of  State,  interagency  organizations, 
regional  centers  like  the  Marshall  Center  in  Europe) 

4 

1 

4 

0 

Ability  to  establish  key  contacts  within  PN  government, 
military,  etc. 

5 

2 

5 

0.5 

Ability  to  make  repetitive/persistent  visits  with  PN 

4 

2 

4 

1 

MSAS  actions  are  integrated  with  other  U.S.  Government 
agency  actions  (e.g.  State  Department,  U.S.  Agency  for 
International  Development,  other  military  units) 

4 

1.25 

4 

0 

Extent  to  which  MSAS  accomplished  policy  objectives  (e.g. 
GCC  end  states  met) 

4 

2 

4 

1 

Notable  improvement/development  of  key  relationships 
with  PN  (e.g.  within  government,  military) 

4 

2 

4 

0.25 

Ability  to  identify  and  recruit  personnel  for  MSAS  (e.g. 
locate,  train,  develop,  track/place)  and  seek  to  establish  a 
formal  career  track  and  career  progression/promotion 
opportunities 

4 

2 

4 

1.25 

Ability  to  develop  plans,  milestones,  projects  the  PN  can 
accomplish  within  realistic  timeframe 

4 

1.25 

4 

0.5 
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Table  4.  Summary  of  Median  and  IQR  Scores  (cont.) 


What  reporting  mechanism  would  you  recommend  the 
MSAS  use  to  track  and  report  their  unit  effectiveness? 

Round  2 

Median 

Round  2 

IQR 

Round  3 

Median 

Round  3 

IQR 

After  Action  Reports 

5 

1 

5 

0 

Survey  results  from  GCC,  Component  staff,  Country  team, 
PN 

4 

1 

4 

0 

Report  unit  mission  readiness  monthly  through  Status  of 
Resources  and  Training  System  (SORTS) 

3 

0.5 

3 

0 

Make  entries  into  Joint  Lessons  Learned  Information 

System  (JLLIS) 

3 

1 

3 

0 

Submit  annual/semi-annual  reports  to  GCC,  Component 
staff,  and  Air  Mobility  Command  through  MSAS  chain  of 
command 

4 

1.25 

4 

1.25 

Track  Partner  Nation  progress  following  MSAS 
engagements  using  a  spreadsheet  and  numeric  rating 
scale  (e.g.  1  -  5  scale:  (0)  -  No  capability  and  (5)  -  Can 
complete  without  assistance) 

4 

2 

4 

1 

Make  entries  into  Geographic  Combatant  Commander 
(GCC)/Air  Force  Theater  Security  Cooperation 

Management  Information  System  (TSCMIS) 

3 

1.5 

3 

1 

Analysis  of  Results 

Summary  of  Group  Participation  and  Outcome 

The  group’s  participation  throughout  the  study  was  noteworthy.  Researchers 
have  reported  that  participation  rates  from  40%  to  75%  can  be  anticipated  with  the  Delphi 
study  (Gordon,  n.d.).  After  three  rounds,  this  study  concluded  with  an  overall  98.3% 
participation  rate  (Table  4). 


Table  5.  Summary  of  Participation 


Round 

#  Invited  to  Participate 

#  Participated 

Participation  Rate  (%) 

1 

20 

19 

95 

2 

20 

20 

100 

3 

20 

20 

100 

Overall  Participation 

98.3 

The  panel  members  identified  15  criteria  they  deemed  important  for  the  MSAS  to 
consider  using  to  measure  their  effectiveness  at  accomplishing  their  BPC  mission 
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(research  question  1).  The  criteria  are  listed  in  Table  6  in  order  of  importance  as  rated  by 
the  group: 


Table  6.  Research  Question  1  Final  Criteria 

What  minimum  criteria  would  you  recommend  the  MSAS  use  to  measure  their 
effectiveness  at  accomplishing  their  Building  Partnership  Capacity  mission? 


5  =Very  Important,  4=lmportant,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 


Criteria 

Group 

Median 

IQR 

Criteria 

1 

5 

0 

MSAS  contributed  towards  Partner  Nation's  (PN)  increased  capability 
and  capacity  (e.g.  response  during  crisis,  stable  or  increasing  mission 
capability  rate,  sustained  personnel  training/task  proficiency, 
interoperability  with  U.S.)  as  reported  by  country  team,  GCC  staff 

2 

5 

0 

MSAS  engagements  are  in  support  of  U.S.  National  Security  policy, 
Geographic  Combatant  Commander  (GCC)  and  Component  Plans 
(e.g.  Theater  Security  Cooperation  Plans,  Campaign  Plans,  and  Air 

Force  Component  Campaign  Support  Plan)  and  Ambassador  Country 
Plans  (e.g.  Mission  Strategic  and  Resource  Plan) 

3 

5 

0 

Ability  to  assess  PN  capability /capacity  (e.g.  access  to  information, 
facilities,  airfields,  equipment,  etc.) 

4 

5 

0.5 

Ability  to  establish  key  contacts  within  PN  government,  military,  etc. 

5 

5 

1 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g. 
language/culture  familiarization,  general  advising  skills,  etc) 

6 

4 

0 

MSAS  actions  are  integrated  with  other  U.S.  Government  agency 
actions  (e.g.  State  Department,  U.S.  Agency  for  International 
Development,  other  military  units) 

7 

4 

0 

Frequency  and  duration/longevity  of  interactions  with  PN 

8 

4 

0 

Requestor  and  types  of  requests  received  (e.g.  Assess,  Train,  Advise, 
Assist,  Equip) 

9 

4 

0 

Ability  to  develop  internal  processes  that  support  MSAS  mission  (e.g. 
record  keeping  for  information  continuity) 

10 

4 

0 

Ability  to  develop  key  relationships  (e.g.  other  military  services, 
Department  of  State,  interagency  organizations,  regional  centers 
like  the  Marshall  Center  in  Europe) 

11 

4 

0.25 

Notable  improvement/development  of  key  relationships  with  PN 
(e.g.  within  government,  military) 

12 

4 

0.25 

Ability  to  develop  achievable  Mission  Essential  Task  Lists  and 
performance  standards  for  PN 

13 

4 

0.5 

Ability  to  develop  plans,  milestones,  projects  the  PN  can  accomplish 
within  realistic  timeframe 

14 

4 

1 

Extent  to  which  MSAS  accomplished  policy  objectives  (e.g.  GCC  end 
states  met) 

15 

4 

1 

Ability  to  make  repetitive/persistent  visits  with  PN 
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The  panel  members  identified  six  criteria  they  considered  important  for  the 
MSAS  as  they  track  and  report  their  effectiveness  at  accomplishing  their  BPC  mission 
(research  question  2).  The  criteria  are  listed  below  in  Table  7  in  order  of  importance  as 
rated  by  the  group: 


Table  7.  Research  Question  2  Final  Criteria 

What  reporting  mechanism  would  you  recommend  the  MSAS  use  to  track  and  report 
their  unit  effectiveness? 


5  =Very  Important,  4=lmportant,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 


Criteria 

Group 

Median 

IQR 

Criteria 

1 

5 

0 

After  Action  Reports 

2 

4 

0 

Survey  results  from  GCC,  Component  staff,  Country  team,  PN 

3 

4 

1 

Track  Partner  Nation  progress  following  MSAS  engagements  using  a 
spreadsheet  and  numeric  rating  scale  (e.g.  1-5  scale:  (0)  -  No 
capability  and  (5)  -  Can  complete  without  assistance) 

4 

3 

0 

Make  entries  into  Joint  Lessons  Learned  Information  System  (JLLIS) 

5 

3 

0 

Report  unit  mission  readiness  monthly  through  Status  of  Resources 
and  Training  System  (SORTS) 

6 

3 

1 

Report  unit  mission  readiness  monthly  through  Status  of  Resources 
and  Training  System  (SORTS) 

Items  of  Importance 

Of  the  16  total  criteria  identified  as  minimum  criteria  the  MSAS  could  use  to 
measure  their  effectiveness  at  accomplishing  their  BPC  mission  (research  question  1), 
three  emerged  as  decidedly  important  in  that  they  attained  the  highest  possible  median 
score  of  5.0  (5.0=Very  Important)  and  greatest  degree  of  group  consensus  with  an  IQR  of 
0.0: 

1.  MSAS  contributed  towards  Partner  Nation ’s  (PN)  increased  capability  and  capacity 
(e.g.  response  during  crisis,  stable  or  increasing  mission  capability  rate,  sustained 
personnel  training/task  proficiency,  interoperability  with  U.S.)  as  reported  by  country 
team,  GCC  staff; 
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Ratings  of  importance  for  this  criterion  ranged  from  4  to  5,  and  19  of  20  panelists 

(95%)  rated  this  item  as  a  5.  One  panelist  stated,  “This  is  why  advisor  units  exist. .  .to 

teach  others  to  fish  so  they  can  feed  themselves.”  Another  panelist  stated: 

. . .  The  only  measure  of  effectiveness  in  BPC  is  to  what  extent  the  capabilities  and 
capacities  of  the  PN  have  been  built.  If  the  GCC  has  given  me  a  mission  of 
building  X’s  capacity  to  organize  and  marshal  cargo  so  that  X’s  can  be  used  to 
effectively  execute  this  task  for  the  purposes  of  humanitarian  support  in  the 
region,  but  I  have  been  unable  to  do  that,  then  I  have  failed.  My  measure  of 
effectiveness  is  zero  no  matter  how  many  key  contacts  I  have  made.  Only  in  the 
initial  stages  of  BPC  does  establishing  key  contacts  have  any  relevance. 

However,  [with  respect  to]  building  capacity,  it  means  little. . . 


2.  MSAS  engagements  are  in  support  ofU.S.  National  Security  policy ,  Geographic 
Combatant  Commander  (GCC)  and  Component  Plans  (e.g.  Theater  Security  Cooperation 
Plans,  Campaign  Plans,  and  Air  Force  Component  Campaign  Support  Plan)  and 
Ambassador  Country  Plans  (e.g.  Mission  Strategic  and  Resource  Plan); 

Ratings  of  importance  for  this  criterion  primarily  ranged  from  4  to  5,  and  18  of  20 
panelists  (90%)  rated  this  item  as  a  5.  Interestingly,  one  panelist  gave  a  rating  of  1  and 
stated,  “This  is  a  statement.  I  can’t  measure  it.”  Another  panelist  stated,  “The  one  and 
only  reason  you  advise  is  to  advance  [United  States  Government]  national  objectives.”  A 
final  comment  from  the  panel  warns  that  “this  issue  is  likely  to  be  the  most  convoluted 
and  challenging  but  linking  all  this  together  will  yield  the  highest  [return  on  investment] 
per  BPC  mission.” 


3.  Ability  to  assess  PN  capability /capacity  (e.g.  access  to  information,  facilities, 
airfields,  equipment,  etc.). 

Ratings  of  importance  for  this  criterion  ranged  from  3  to  5,  and  15  of  20  panelists 
(75%)  rated  this  item  as  a  5.  One  panel  member  said,  “Absolutely  [the]  critical  first 
step.” 
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Of  the  seven  total  criteria  identified  as  the  reporting  mechanism  the  MSAS  could 
use  to  track  and  report  their  unit  effectiveness  (research  question  2),  one  emerged  as 
decidedly  important  in  that  it  attained  the  highest  possible  median  score  of  5.0  (5.0  = 
Very  Important)  and  greatest  degree  of  group  consensus  with  an  IQR  of  0.0: 

1.  After  Action  Reports  (AAR). 

Ratings  of  importance  for  this  criterion  ranged  from  4  to  5,  and  18  of  20  panelists 
(90%)  rated  this  item  as  a  5.  Panel  members  commented,  “AARs  will  be  critical  in  the 
continued  assessment  process... appropriately  edited  reports  must  be  forwarded  to  [higher 
headquarters]  to  ensure  support  and  understanding  of  MSAS  actions,  accomplishments 
and  future  support  requirements;”  and  “[AARs]  are  the  key  document  for  tracking  the 
training  provided  to  PN  and  results.” 

Items  That  Did  Not  Reach  Consensus 

There  were  two  items  that  did  not  reach  consensus  (an  IQR  of  1  or  less) — one 
from  each  of  the  criteria  the  group  identified  for  research  questions  1  and  2.  However, 
stability  was  reached  on  both  items  (a  median  value  change  of  .75  or  less  between 
rounds).  Both  criterions  attained  a  group  median  score  of  4.0  (4.0  =  Important)  and  an 
IQR  of  1.25: 

1.  Ability  to  identify  and  recruit  personnel  for  MSAS  (e.g.  locate,  train,  develop, 
track/place)  and  seek  to  establish  a  formal  career  track  and  career 
pr  ogres  si  on/promo  ti on  opportune ti  es; 

Ratings  of  importance  for  this  criterion  ranged  from  1  to  5,  and  5  of  20  panelists 
(25%)  rated  this  item  as  a  5.  While  consensus  was  not  reached  (IQR  was  1 .25),  rating 
stability  of  4  was  maintained  between  rounds  2  and  3,  while  IQR  was  reduced  from  2  to 
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1.25.  This  change  serves  as  an  indicator  of  convergence  of  opinion  as  the  panel  moved 
closer  to  consensus  on  this  item. 

2.  Submit  annual/ semi-annual  reports  to  GCC,  Component  staff,  and  Air  Mobility 
Command  through  MSAS  chain  of  command. 

Ratings  of  importance  for  this  criterion  ranged  from  3  to  5,  and  5  of  20  panelists 
(25%)  rated  this  item  as  a  5.  Similarly,  while  consensus  was  not  reached  (IQR  was  1.25), 
rating  stability  of  4  with  an  IQR  of  1.25  was  maintained  between  rounds  2  and  3. 

While  the  Delphi  allowed  the  panel  of  experts  to  use  their  collective  “voice”  to 
come  to  consensus  and  detennine  the  final  list  of  criteria,  it  is  important  to  note  that 
outcome  of  the  Delphi  is  simply  an  opinion  and  “only  as  valid  as  the  opinions  of  the 
experts  who  made  up  the  panel”  (Yousuf,  2007:5).  Nonetheless,  by  using  the  Delphi 
method,  the  expert  panelists’  inputs  were  statistically  summarized,  thereby  assigning  a 
quantitative  value  to  their  opinions.  In  addition  to  quantifying  the  panel’s  opinion,  this 
study  also  was  able  to  gain  unanimous  consensus  on  the  majority  of  criteria  the  panel 
identified.  The  next  chapter  offers  a  conclusion  to  this  study  and  provides 
recommendations  for  further  research. 
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V.  Conclusion  and  Recommendations 


Conclusions  of  Research 

The  purpose  of  this  research  effort  was  to  develop  criteria  the  MSAS  could  use  to 
measure,  track,  and  report  their  effectiveness  at  accomplishing  their  BPC  mission. 
Utilizing  the  Delphi  method,  a  panel  of  experts  was  assembled  who  agreed  upon  a  final 
list  of  criteria.  From  the  original  66  items  identified  in  the  first  round  of  the  study, 
consensus  was  reached  on  1 5  criteria  for  research  question  1  (What  minimum  criteria 
would  you  recommend  the  MSAS  use  to  measure  their  effectiveness  at  accomplishing 
their  Building  Partnership  Capacity  mission?)  and  6  criteria  for  research  question  2  (What 
reporting  mechanism  would  you  recommend  the  MSAS  use  to  track  and  report  their 
unit  effectiveness?).  While  the  panel,  through  their  unifonn  opinion,  identified  minimum 
criteria  the  MSAS  could  use  to  measure  their  effectiveness,  the  results  from  this  study 
only  serve  as  a  starting  point  for  the  ongoing  discussion  required  for  the  proper 
development  and  tasking  of  AAs  and  measuring  their  impact  in  support  of  the  BP/BPC 
mission. 

A  literature  review  of  non-military  organizations  that  seek  to  build  partner 
capacity  such  as  the  Peace  Corps  and  non-governmental  agencies  has  identified  an 
ongoing  effort  to  determine  how  best  to  track,  measure,  and  report  the  effects  of  their 
activities.  Teichert  (2008)  noted  that  the  Peace  Corps,  46  years  after  beginning  its 
mission,  did  not  establish  a  fonnal  process  by  which  to  assess  effectiveness  until  2007  via 
the  institution  of  the  Office  of  Strategic  Infonnation,  Research,  and  Planning.  Prior  to 
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this  action,  the  Peace  Corps  primarily  relied  on  volunteer  exit  surveys  as  a  means  for 
indication  of  their  effectiveness  and  not  feedback  from  those  they  served. 

Gene  Ogiogio  (2005)  of  the  African  Capacity  Building  Foundation  acknowledges 
that  good  measures  are  essential  to  developing  a  reliable  and  adaptable  evaluation  system 
for  capacity  building.  The  challenge  he  identifies  is  that  the  collective  experience  for 
evaluating  effectiveness  is  still  growing,  and,  as  a  result,  viable  methodological 
frameworks  and  instruments  are  not  readily  available.  Ogiogio  (2005:2)  concludes  by 
stating,  “The  scale  and  complexity  of  the  capacity  building  process  coupled  with  the 
difficulty  of  impact  assessment  in  the  field  make  the  development  of  performance 
measures  and  evaluation  systems  all  the  more  challenging.” 

To  further  support  this  point,  a  2009  RAND  study  highlighted  the  lack  of  a 
comprehensive  assessment  methodology  for  DOD  security  cooperation  efforts.  The 
“dispersed  and  long-term”  nature  of  conducting  security  cooperation  and  the  lack  of 
standardization  or  shared  “assessment  language”  has  resulted  in  the  inability  to  make 
appropriate  resourcing  decisions.  The  study  claims  that  there  are  efforts  underway  to 
remedy  this  deficiency.  However,  no  date  or  timeline  is  given.  They  conclude  by  saying 
that  a  key  goal  will  ultimately  be  to  infuse  more  objectivity  into  the  assessment  process 
through  the  development  of  a  common  security  cooperation  assessment  framework 
(Moroney  et  ah,  2009). 

In  spite  of  this  shared  challenge  between  the  various  capacity  building-focused 
organizations,  a  means  of  measuring  impact,  preferably  quantitatively,  is  often  required 
in  order  for  decision  makers  to  justify,  plan,  obligate,  and  properly  account  for  resource 
expenditures.  The  use  of  metrics  is  one  tool  commonly  utilized  to  convey  effectiveness. 
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This  is  especially  true  within  AMC  where  data  such  as  fleet  aircraft  availability  rates  and 
overall  fuel  consumption  are  closely  tracked.  Similarly,  the  tactical-level  actions  each 
MS  AS  takes  with  their  assigned  PNs  must  be  able  to  be  conveyed  in  some  manner  that 
provides  awareness  at  how  the  overall  engagement  is  going  and  overall  effectiveness  of 
outcomes.  This  is  important  because  tactical-level  outcomes  should  support  the  strategic- 
level  end  states  as  determined  by  AMC,  the  geographic  combatant  commander  (GCC), 
the  U.S.  embassy  country  teams,  and  even  the  President.  If  done  correctly,  outcomes 
may  result  in  country  access,  overfly  rights,  and  coalition  support  during  humanitarian 
assistance/disaster  response  activities.  For  the  time  being,  until  a  practical  methodology 
is  developed  and  fielded  within  the  DOD,  acceptance  of  qualitative,  intuitive,  and 
subjective  feedback  may  need  to  continue  to  be  the  standard. 

Significance  of  Research 

This  study  assembled  a  diverse,  highly-motivated  expert  panel  of  20  individuals 
and  formally  captured  group  opinion  to  identify  important  criteria  the  MSAS  could  use  in 
accomplishing  its  BPC  mission.  Through  the  use  of  the  Delphi  technique,  a  98.3% 
response  rate  was  achieved  across  three  rounds  of  questionnaires  and  feedback.  This 
achievement  alone  is  especially  noteworthy  since  response  rates  reported  in  other  Delphi 
studies  ranged  between  40-75%  (Gordon,  n.d.).  For  non-Delphi  studies,  response  rates  to 
an  emailed  survey  can  range  from  40-60%  (40%  average,  50%  good,  60%  very  good) 
(Instructional  Assessment  Resources,  n.d.).  Additionally,  panel  members  included  a 
General  officer,  multiple  field-grade  officers,  senior  non-commissioned  officers,  service 
civilians,  and  a  retired  State  department  diplomat.  They  represented  HAF,  AMC,  AETC, 
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AFSOC,  CRW,  MSAS,  base-level,  and  deployed  AAs  from  Afghanistan.  Collectively, 
they  brought  a  combined  33  years  of  PN  advisory  experience  to  the  study.  As  a  result  of 
their  positions  and  experiences,  the  panelists  brought  varying  perspectives  to  each  Delphi 
round.  From  their  inputs,  feedback,  and  ratings,  their  group’s  “voice”  shaped  the  final 
list  of  criteria.  Furthennore,  several  of  the  panel  members,  as  key  stakeholders,  are  in 
positions  that  can  utilize  these  findings  as  they  develop  and  mold  the  future  of  the  AF’s 
BP/BPC  mission.  In  combination,  all  of  these  points  can  be  taken  as  evidence  as  to  the 
quality  and  validity  of  this  study  and  the  importance  of  this  topic. 

Noteworthy  Advancements  towards  Developing  the  AF  BP/BPC  Program 

During  the  course  of  this  study,  significant  initiatives  were  undertaken  within  the 
AF’s  BP/BPC  realm  to  address  previously  identified  challenges.  These  include  the 
following: 

HAF 

In  February  2012,  CSAF  Schwartz  signed  and  published  the  initial  USAF  Air 
Advising  Operating  Concept  which  applies  to  all  AF  personnel  and  “establishes  a 
common  framework  and  guidelines  for  planning  and  conducting  USAF  air  advising 
activities  in  support  of  geographic  commander  (GCC)  requirements”  (Department  of  the 
Air  Force,  2012:v).  This  comprehensive  document  effectively  outlines  the  various  facets 
that  comprise  air  advising  and  provide  key  information  on  how  to  utilize  this  function. 
Additionally,  HAF/A9A  has  been  tasked  by  CSAF  via  Planning  Order  for  the  USAF 
Campaign  Support  Plan  0800-12  to  develop  a  security  cooperation  (SC)  framework 
which  will  be  a  “collection  of  agreed  upon  definitions,  taxonomies,  elements,  criteria, 
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standardized  data  requirements,  reporting  methods  and  formats  that  shape  how 
assessments  for  SC  are  accomplished  and  reported”  (Department  of  the  Air  Force, 
2012a:slide  4). 

Major  Commands 

AMC  has  fonnalized  and  published  a  design  operational  capability  statement  for 
each  MSAS  and  is  currently  developing  mission  essential  task  lists  against  which  the 
MSAS  will  be  assessed.  AETC,  as  the  service  core  function  lead  integrator  for  BP/BPC, 
readily  supported  the  maturing  MSAS  mission  by  utilizing  the  Air  Advisor  Academy  to 
develop  and  instruct  courses  specifically  tailored  for  the  GPF  Ainnen  tasked  and  assigned 
to  the  MSAS. 

MSAS 

In  an  ongoing  effort,  each  MSAS  is  preparing  for  potential  taskings.  Specifically, 
all  of  the  required,  in-garrison  organizing,  training,  and  equipping  responsibilities  along 
with  measuring  and  reporting  unit  readiness  monthly  are  accomplished  in  accordance 
with  established  service  standards.  Each  MSAS  is  ensuring  their  personnel  are  receiving 
appropriate  AA  training,  functional  skill  training,  and  career  development.  They  are  also 
working  to  recruit  and  source  personnel  to  fill  position  vacancies.  Additionally,  both 
squadrons  have  had  the  opportunity  to  make  visits  to  their  areas  of  responsibility.  The 
57 1  MSAS  spent  a  month  (January  -  February  2012)  in  Honduras.  Their  AAs  conducted 
training  and  assisted  the  Honduran  Air  Force  with  installing  radar  altimeters  in  some  of 
their  helicopters.  In  March  2012,  the  818  MSAS  participated  in  the  inaugural  African 
Partnership  Flight  event  in  Ghana.  This  event  was  a  2-week,  military-to-military 
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familiarization  and  training  opportunity  between  the  Ainnen  from  the  U.S.,  Ghana,  Togo, 
Benin,  Nigeria,  and  Senegal. 


Recommendations  for  Action 

With  regard  to  the  progress  being  made  and  efforts  being  undertaken  at  each 
organizational  level,  the  findings  from  this  study  should  be  taken  into  consideration  to 
ensure  the  key  criteria,  as  highlighted  by  the  Delphi  subject  matter  experts,  is  not 
overlooked  when  developing  the  necessary  capabilities  required  to  achieve  the  desired 
effects  the  MSAS  is  established  to  generate. 

Referencing  the  USAF  Air  Advising  Operating  Concept  (2012b),  the  following 
items  should  be  further  addressed  jointly  by  each  echelon: 

•  The  ability  to  develop  air  advising  strategy,  policy,  and  procedures; 

•  The  ability  to  identify  and  document  air  advising  requirements  that  support 
Guidance  for  the  Employment  of  the  Force  end  states  and  GCC  Theater 
Campaign  Plans; 

•  The  ability  to  accomplish  comprehensive  and  deliberate  theater-level  planning  of 
air  advising  activities; 

•  The  ability  to  grasp  the  intricacies  of  the  authorities  and  funding  available  to 
support  air  advising  activities  and  to  apply  proper  authorities  and  funding  to 
specific  activities; 

•  The  ability  to  establish  and  use  measures  of  effectiveness  or  measures  of 
perfonnance  (MOE/MOP)  for  partner  nation  activities; 

•  The  ability  to  identify  the  specific  resources  required  for  an  air  advising  activity 
and  to  detennine  the  most  appropriate  method  of  obtaining  these  resources; 

•  The  ability  to  develop  and  execute  an  effective  strategic  communications  plan; 

•  The  ability  to  perfonn  an  after-action  assessment  of  a  partner  nation  engagement 
and  to  provide  feedback  of  lessons  learned  to  involved  U.S.  and  partner  nation 
personnel  through  the  security  cooperation  organization  (SCO). 
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Based  on  the  Delphi  panel’s  comments  and  information  gained  through  this  study, 
five  recommendations  are  provided.  First,  in  an  ongoing  effort  to  mature  its  readiness 
and  operational  capability,  each  MSAS  could  continue  working  with  and  learning  from 
the  6  SOS  combat  aviation  advisors  (CAAs).  Currently,  as  the  most  experienced  and 
developed  team  of  advisors  within  the  AF,  the  CAAs  have  been  working  with  selected 
PNs  around  the  globe  since  1994.  During  this  time,  they  have  established  repeatable 
processes  and  procedures  that  have  been  tested  and  found  to  deliver  results.  By 
referencing  and  modeling  6  SOS  operating  instructions,  checklists,  and  after  action 
reports,  as  appropriate,  the  MSAS  can  use  the  experience  and  knowledge  the  6  SOS  has 
gained  to  shape  their  own  internal  processes  and  procedures  by  which  to  accomplish  their 
BPC  mission.  In  essence,  instead  of  “reinventing  the  wheel,”  the  MSAS  can  modify  it  to 
fit  their  needs  as  required. 

Second,  HAF,  SAF/IA,  and  MAJCOM  staffs  (e.g.  A3,  A5,  A 9,  etc.)  must  be 
deliberately  targeted  and  better  educated  on  what  the  MSAS  are  designed  to  do  and 
capable  of  accomplishing.  These  staffs  are  in  the  best  position  to  help  the  MSAS  attain 
access  to  GCCs,  country  teams,  and,  ultimately,  to  PNs.  This  is  also  in  alignment  with 
the  top  three  criteria  the  Delphi  panel  identified,  which  were  MSAS  contribution  to  PN 
increased  capability;  MSAS  engagements  support  national  security,  GCC  and/or 
Ambassador  plans;  and  MSAS  assessment  of  PN  capability  and  capacity. 

Perhaps  the  linchpin  to  the  MSAS  ability  to  effect  positive  change  or  effect 
towards  their  BPC  mission  rests  with  the  NAF.  As  the  service  component’s  interface  to 
the  GCC  and  U.S.  embassy  country  team,  the  NAF  serves  as  primary  integrator  for 
promoting,  lobbying,  and  obligating  the  MSAS  to  engage  specified  PNs  in  support  of 
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strategic  goals.  It  is  essential  that  the  NAF  be  fully  informed  on  the  MS  AS  so  they  can 
properly  represent  their  capabilities  during  planning  conferences  or  during  the  initial 
stages  of  humanitarian  assistance/disaster  response  missions.  Moreover,  the  NAF  can 
also  serve  as  the  feedback  conduit  for  the  MSAS  regarding  the  engagements  they  were 
tasked  to  undertake.  Nothing  can  happen  without  a  tasking  or  support  request.  The  NAF, 
oftentimes,  is  in  the  best  position  to  advise  potential  MSAS  “customers”  and  help  with 
the  tasking  generation. 

Third,  AMC  and  the  MSAS  could  continue  to  craft  and  execute  a  strategic 
communications  plan.  Through  deliberate  efforts  to  incorporate  base,  MAJCOM,  HAF, 
and  GCC  public  affairs  offices  and  related  media  outlets  into  their  operations,  the  MSAS 
can  create  awareness  and  interest  of  their  unique  mission  set  and,  in  turn,  generate 
support  requests.  Utilizing  modern  social  media  like  YouTube  or  Facebook  sites  is  an 
inexpensive  promotional  opportunity.  Additionally,  the  MSAS  can  seek  further  venues 
to  endorse  the  BP/BPC  mission  via  professional  associational  conferences  such  as  the 
Airlift/Tanker  Association,  the  Logistics  Officer  Association,  the  AF  Association,  etc. 
Further  awareness  can  be  provided  through  professional  military  education  (Ainnan 
Leadership  School,  Squadron  Officer  School,  NCO  Academy,  etc.).  These  settings  could 
generate  interest  from  the  field,  gamer  senior  leadership  support,  or  target  skill  sets  the 
squadrons  would  be  looking  to  recruit. 

Fourth,  AMC  and  AETC  could  continue  working  to  bolster  and  refine  the  Air 
Advisor  course  taught  at  the  Air  Advisor  Academy  at  Joint  Base  McGuire-Dix- 
Lakehurst,  NJ.  For  example,  incorporating  guest  speakers  from  the  MSAS  who  have 
recently  completed  a  PN  engagement  can  help  to  inform  and  educate  the  current  students 
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on  what  to  expect  during  their  tenure.  It  would  also  keep  course  instructors  and 
administrators  current  on  the  challenges,  successes,  and  areas  needing  improvement 
within  BP/BPC. 


Fifth,  AAR  findings  could  be  entered  into  AF  Theater  Security  Cooperation 
Management  Information  System  (AF  TSCMIS)  following  a  PN  engagement.  This 
coincides  with  the  panel’s  recommendation  to  use  action  reports  (AARs)  as  the  number  1 
mechanism  to  track  and  report  effectiveness.  As  directed  by  SECAF  Donley  and 
overseen  by  Secretary  of  the  Air  Force/Intemational  Affairs  office,  AF  TSCMIS  is  the 
tool  for  the  AF  to  track  and  assess  security  cooperation  activities  (Department  of  the  Air 
Force,  2009a).  This  requirement  is  critical  since  the  AF  TSCMIS  data  is  used  to  help  the 
GCC  track  the  various  security  cooperation  activities  that  occur  in  their  area  of 
responsibility  as  well  as  to  help  justify  future  planning  and  resourcing  decisions.  Panel 
members  commented  as  follows: 

•  “[TSCMIS]  is  instrumental  to  validating  and  justifying  the  MSAS.” 

•  “[It]  can  be  used  to  synergize  efforts  with  other  services  and  agencies  as  they  have 
visibility  into  TSCMIS  information.” 

•  “This  is  the  way  to  track  the  future. .  .and  is  the  best  means  to  build  credibility  and 
[money]  for  a  dedicated  BP/BPC  GPF.” 

By  heeding  the  Delphi  panel’s  opinion  and  acting  upon  these  five 
recommendations,  the  expected  outcomes,  when  pooled  with  the  results  of  other  security 
cooperation  efforts,  may  help  establish  a  common  assessment  framework.  This 
foundation  could  allow  outcomes  to  be  objectively  captured  and  quantitatively  reported. 
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Limitations  of  the  Study 

Linstone  and  Turoff  (2002)  note  that  for  as  many  individuals  who  have  had 
success  with  conducting  a  Delphi  study,  there  are  as  many  who  have  had  unfavorable 
results.  They  highlight  that  there  are  five  common  reasons  for  failure  (2002:6): 

•  Imposing  monitor  views  and  preconceptions  of  a  problem  upon  the  respondent 
group  by  over-specifying  the  structure  of  the  Delphi  and  not  allowing  for 
contribution  of  other  perspectives  related  to  the  problem. 

•  Assuming  that  Delphi  can  be  a  surrogate  for  all  other  human  communications  in  a 
given  situation. 

•  Poor  techniques  of  summarizing  and  presenting  the  group  response  and  ensuring 
common  interpretations  of  the  evaluation  scales  utilized  in  the  exercise. 

•  Ignoring  and  not  exploring  disagreement  so  that  discouraged  dissenters  drop  out 
and  an  artificial  consensus  is  generated. 

•  Understanding  the  demanding  nature  of  a  Delphi  and  the  fact  that  the  respondents 
should  be  recognized  as  consultants  and  properly  compensated  for  their  time  if  the 
Delphi  is  not  an  integral  part  of  their  job  function. 

Additionally,  there  are  three  common  limitations  associated  with  Delphi  studies. 
First,  the  data  produced  from  a  Delphi  study  is  opinion-based  and  not  evidence-based  or 
irrefutable  fact.  From  this  point  alone,  many  have  dismissed  the  usefulness  of  this 
method.  Sackman  (1974:11),  a  noted  critic  of  the  method,  writes,  “If  Delphi  is  to  be 
treated  seriously  as  a  professional  technique,  it  must  be  judged  by  basic,  minimum 
standards  applicable  to  all  empirical  social  science.”  He  further  notes  that  Delphi  lacks 
reliability  measurements  and  that  the  findings  should  be  able  to  be  scientifically 
validated. 

Second,  there  are  no  universally  recognized  standards  and  guidelines  for 
organizing  and  conducting  a  Delphi  study  (Keeney  et  al.,  2006:208).  For  example,  a 
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literature  review  reveals  that  there  are  a  variety  of  opinions  on  topics  such  as  how  panel 
members  should  be  identified,  how  many  panelists  should  be  used,  how  many  rounds 
should  be  conducted,  how  consensus  should  be  defined,  how  data  should  be  analyzed  and 
reported,  and  when  a  study  should  be  concluded  (Powell,  2003:378-381;  Sumsion, 
1998:154;  Hsu  and  Sandford,  2007:3-5;  Keeney  et  ah,  2006:208-21 1).  Based  on  this  lack 
of  standardization,  a  considerable  amount  of  time  has  to  be  invested  up  front  conducting 
a  literature  review  in  order  to  identify  how  best  to  identify  the  appropriate  thresholds  for 
assembling  a  study.  Ultimately,  it’s  up  to  the  researcher  which  criterion  will  be  most 
useful  when  designing  their  particular  study. 

Third,  conducting  a  Delphi  study  can  be  time-consuming.  To  properly  prepare, 
time  must  be  allotted  to  carryout  interviews  and  design,  distribute,  collect,  analyze,  and 
record  initial  and  subsequent  round  questionnaires.  Delbecq  et  al.  (1975)  recommend 
that  45  days  be  allotted  to  conduct  a  study.  They  also  suggest  two  weeks  be  given  to 
panel  members  to  respond  during  each  round.  However,  Keeney  et  al.  (2006)  reviewed 
multiple  studies  and  found  that,  while  each  one  was  well  planned  and  executed,  authors 
underestimated  the  amount  of  time  it  would  take  to  gain  consensus.  They  reported 
studies  ran  from  4-16  months  to  complete.  Lack  of  panel  participation  often  required 
follow-up  contact  by  way  of  reminder  e-mails  and  phone  calls,  thus  adding  to  the  overall 
time  required. 

This  particular  study  was  also  limited  by  time.  Specifically,  the  researcher 
worked  against  a  compressed  academic  calendar  and  class  schedule  to  design,  administer, 
and  conclude  the  study  while  leaving  enough  time  to  properly  draft  the  research  paper.  If 
more  time  were  available,  another  round  could  be  conducted  or  interviews  carried  out  to 
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further  refine  the  expert  panel’s  thoughts  regarding  the  final  list  of  criteria  and  its 
potential  uses  towards  BP/BPC.  However,  in  spite  of  these  limitations,  Gordon  (n.d.:  10) 
claims  that  “No  better  way  exists  to  collect  and  synthesize  opinions  than  Delphi.” 
Skulmoski  (2007)  further  hails  that  this  method  is  mature  and  highly  adaptable  and  used 
by  a  variety  or  researchers  all  over  the  world. 

Recommendations  for  Future  Research 

As  mentioned  earlier,  the  results  from  this  study  only  serve  as  a  starting  point. 
Ongoing  discussion  is  required  to  further  the  development  of  an  assessment  framework 
for  measuring,  tracking,  and  reporting  AA  effectiveness  towards  BP/BPC.  To  this  end, 
there  are  three  recommendations  for  future  research.  First,  referencing  the  criteria 
identified  in  this  study,  follow-on  interviews  could  be  conducted  with  a  sample  of  the 
panel  members  in  order  to  further  explore  the  top-rated  criteria  and  how  to  further  refine 
them  into  more  specific  tasks  and  actions. 

Second,  in  conjunction  with  HAF/A9  and  SAF/IA,  a  Policy  Delphi  study  could  be 
conducted  to  assist  with  the  development  of  an  assessment  framework  for  the  AF 
Security  Cooperation  Campaign  Support  Plan.  A  Policy  Delphi  is  an  organized  method 
for  comparing  views  and  information  regarding  to  a  specific  policy  area  and  allows  panel 
members  with  varying  opinions  to  address  and  assess  differing  viewpoints  (Turoff, 

2002).  After  the  study  is  concluded,  policy  makers  could  then  use  the  inputs  generated  to 
help  fonn  the  necessary  policy  related  to  Security  Cooperation  and  its  assessment 
framework. 


58 


Third,  if  more  timely  input  is  needed  to  develop  the  assessment  framework, 
HAF/A9,  with  support  of  SAF/IA,  could  fund  and  conduct  a  Nominal  Group  Technique 
(NGT)  study.  The  NGT  is  an  effective  way  to  make  pooled  judgments  or  decisions  in 
groups  which  meet  face-to-face  (Delbecq  et  ah,  1975).  Select  panel  members  identified 
in  this  study  could  be  assembled.  Similar  to  the  goal  of  the  Delphi  study,  they  would 
work  to  generate  ideas  and  develop  a  group  opinion  through  consensus.  The  primary 
benefit  of  NGT  is  that  answers  are  generated  considerably  faster  since  the  panel  is  in  the 
same  room. 

Summary 

In  closing,  as  emphasized  in  the  USAF  Global  Partnership  Strategy  (2008:3), 
“Partnerships  provide  the  force  multiplier  essential  to  the  success  of  winning  today’s 
fight  while  preparing  for  tomorrow’s  challenges.”  Through  MSAS  AAs,  tactical  level 
actors  on  the  world’s  stage  are  able  to  link  tactical  actions  to  strategic  requirements  in 
support  of  global  U.S.  strategic  goals.  Furthermore,  from  their  engagements,  AAs 
increase  the  likelihood  of  PNs  ability  to  benefit  from  formalizing  and  sustaining  an 
aviation  enterprise.  Results  can  lead  to  a  PN’s  ability  to  better  self-govem,  protect  its 
borders,  provide  for  its  population,  expand  economic  potential,  and  serve  as  a  regional 
leader  and  stabilizer.  As  Benjamin  Franklin’s  old  adage  goes,  “An  ounce  of  prevention  is 
worth  a  pound  of  cure.” 

However,  the  imposed  requirement  of  having  to  quantify  or  report  effectiveness  is 
difficult  in  regards  to  relationship-building — especially  during  a  time  of  fiscal 
conservativeness.  While  reporting  inputs,  outputs,  and  outcomes  through  metrics  is 
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common  place  within  the  public  and  private  sector  as  a  form  of  accountability  of 
resources,  a  universal,  one-size-fits-all  assessment  tool  for  quantifying  qualitative-type  of 
outcomes  such  as  trust  and  good  will  has  yet  to  be  developed.  In  regards  to  the  MSAS, 
since  every  PN  is  different,  it  is  possible  for  a  common  assessment  platform  to  be 
developed  and  adjusted  for  use  as  required.  As  mentioned  earlier,  there  are  current 
efforts  working  to  achieve  that  goal.  Until  one  is  fielded,  feedback  such  as,  “Bottomline: 
if  the  [GCC]  and  Ambassadors  are  happy,  we  are  doing  a  good  job”  can  serve  as  a 
practical  indicator  of  effectiveness  (M.  Grub,  personal  communication,  April  9,  2012).  In 
the  end,  when  wanting  to  quantify  things,  it  would  be  wise  to  heed  Albert  Einstein  when 
he  wisely  stated,  "Not  everything  that  counts  can  be  counted,  and  not  everything  that  can 
be  counted  counts."  This  is  not  to  downplay  the  importance  of  quantifying  outcomes  but 
to  understand  that  the  initial  challenge  is  determining  what  to  count. 

The  author  postulates  that  as  the  BP/BPC  program  matures,  as  communication 
and  reporting  mechanisms  improve,  as  the  mission  and  capability  is  better  defined,  and  as 
its  purpose  is  recognized,  the  program  will  reach  its  potential.  The  ultimate  payoff  for 
BP/BPC  investment  is  building  mutually  beneficial  PN  relationships.  Through 
partnerships,  PNs  can  be  equipped  to  independently  address  conflicts  using  their  own 
organic  resources  and  to  manage  humanitarian/disaster  response  with  little  or  no  U.S. 
military  intervention.  Through  purposeful  PN  taskings  aimed  to  support  combatant 
commander  strategic  goals,  the  MSAS,  through  AA’s  as  critical  enablers,  can  help  ensure 
U.S.  national  security  is  attained  and  preserved. 
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Appendix  A 


Mobility  Support  Advisory  Squadron’s  BPC  Mission  Delphi  Study  Questionnaire  #1 

Thank  you  for  agreeing  to  participate  in  this  Delphi  Study.  The  purpose  of  this  research  is  to 
determine  what  appropriate  measurement  criteria  and  reporting  mechanism  the  Mobility  Support 
Advisory  Squadron  could  use  to  track  and  report  unit  effectiveness  toward  their  Building  Partnership 
Capacity  mission.  The  sponsor  for  this  research  is  Colonel  Christopher  A.  Pike,  Vice  Commander,  21st 
Expeditionary  Mobility  Task  Force,  Joint  Base  McGuire -Dix-Lakehurst,  NJ. 

Please  note  the  following: 

Benefits  and  risks:  There  are  no  personal  benefits  or  risks  for  participating  in  this  study.  Your  participation 
in  completing  this  questionnaire  should  take  less  than  1 5  minutes  per  round. 

Confidentiality:  Your  responses  are  completely  confidential,  and  your  identity  will  remain  anonymous. 

No  individual  data  will  be  reported;  only  data  in  aggregate  will  be  made  public.  Data  will  be  kept  in  a 
secure,  locked  cabinet  to  which  only  the  researchers  will  have  access.  If  you  have  any  questions  or 
concerns  about  your  participation  in  this  study,  please  contact: 


JOSEPH  E.  WHITTINGTON,  JR.,  Major,  USAF 

ALAN  R.  HEMINGER,  Ph.D. 

IDE  Student,  Advanced  Study  of  Air  Mobility 

Associate  Professor  of  Management  Information 

USAF  Expeditionary  Center 

Systems 

JB  McGuire -Dix-Lakehurst,  NJ 

Graduate  School  of  Engineering  and  Management 

DSN  312-650-7748 

Air  Force  Institute  of  Technology 

Comm  609-754-7748 

Wright-Patterson  AFB,  OH 

Voice:  937-255-3636  (785-3636  DSN)  ext  7405 

Voluntary  consent:  Your  participation  is  completely  voluntary.  You  have  the  right  to  decline  to  answer 
any  question,  to  refuse  to  participate,  or  to  withdraw  at  any  time.  Your  decision  of  whether  or  not  to 
participate  will  not  result  in  any  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Completion 
of  the  questionnaire  implies  your  consent  to  participate. 

Background:  Because  each  respondent  will  have  a  different  perspective,  allow  me  to  give  a  brief 

overview  of  the  topic  of  study. 

National  strategic-level  guidance  has  outlined  how  the  US  will  continue  to  endeavor  in  “The  Long 
War.”  From  the  President  downward,  emphasis  has  been  placed  on  the  importance  of  the  US  to  work  with, 
by,  and  through  foreign  counterparts  to  ultimately  bolster  US  national  security  interests.  The  US  Air  Force 
has  assumed  the  leadership  role  to  develop  the  aviation  capacity  and  capability  of  selected  partner  nations 
(PN).  The  key  assets  to  accomplishing  this  objective  are  Air  Advisors  (AA).  The  Air  Force  defines  an  AA 
as,  “an  Airman  specially  trained  and  educated  to  apply  aviation  support  and  operational  expertise  to  assess, 
train,  advise,  assist,  and  equip  PNs  in  the  development  and  application  of  their  aviation  enterprise  to  meet 
their  national  needs,  in  support  of  US  interests.”  As  the  worldwide  leader  in  air  mobility  capability,  Air 
Mobility  Command  has  activated  two  Mobility  Support  Advisory  Squadrons  (MSAS)  who  are  preparing 
their  AAs  to  assess,  advise,  train,  assist,  and  equip  selected  PNs  as  tasked  (please  reference  attached 
brochure  for  more  information). 

While  initial,  broad-based  guidance  was  drafted  and  published  in  order  to  assist  timely  unit  stand- 
up  actions,  necessary  in-depth  guidance  has  yet  to  be  developed  and  provided.  One  key  element  that  has 
not  been  addressed  pertains  to  assessment.  Specifically,  no  direction  was  given  on  how  the  MSAS  are  to 
measure  their  effectiveness  at  conducting  the  Building  Partner  Capacity  (BPC)  mission.  What  makes  this 
requirement  even  more  pressing  is  that  establishing,  building,  and  fostering  relationships  with  PNs  are 
time-consuming  and  lengthy  in  nature.  Furthermore,  it  is  difficult  to  quantify  expected  outcomes  such  as 
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trust,  good-will,  and  positive  feelings  towards  the  U  S,  making  measurement  of  outcome,  at  present,  almost 
entirely  subjective  in  nature. 

Therefore,  this  qualitative  research  effort  seeks  to  determine  what  appropriate  measurement 
criteria  and  reporting  mechanism  the  MSAS  could  use  to  track  and  report  unit  effectiveness  toward  BPC. 

By  responding,  you  have  the  unique  opportunity  to  shape  how  the  MSAS  can  tie  their  tactical-level 
actions  to  strategic  outcomes.  The  data  obtained  through  this  study  will  form  recommendations  that  will 
be  offered  to  Air  Force  decision  makers  at  the  Fleadquarters,  Major  Command  (Air  Mobility  Command), 
and  wing  levels  in  order  to  shape  the  future  of  this  mission.  Thank  you  for  participating  in  this  study.  I 
appreciate  your  time  and  candid  responses. 

Process: 

1.  Please  complete  this  questionnaire  electronically  and  return  it  to:  joseph.whittington@us.af.mil  no 
later  than  7  February  2012.  If  you  have  questions,  I  can  be  reached  at  609-754-7748  or  via  DSN  650- 
7748. 

2.  This  questionnaire  is  an  instrument  of  a  Delphi  Study.  The  Delphi  method  is  an  iterative,  group 
communication  process  which  is  used  to  collect  and  distill  the  judgments  of  experts  using  a  series  of 
questionnaires  interspersed  with  feedback.  The  questionnaires  are  designed  to  focus  on  problems, 
opportunities,  solutions,  or  forecasts.  Each  questionnaire  is  developed  based  on  the  results  of  the  previous 
questionnaire.  The  process  continues  until  the  research  question  is  answered.  For  example,  when 
consensus  is  reached,  sufficient  information  has  been  exchanged.  This  usually  takes,  on  average,  3-4 
rounds. 

3.  There  are  three  background  questions  and  two  primary  questions  for  this  round.  The  background 
questions  are  requested  to  establish  your  particular  expertise  for  the  study  and  will  not  be  shared 
specifically  in  the  report.  Again,  the  survey  is  non-attribution,  so  please  elaborate  fully  on  your  answers. 
Once  all  survey  responses  are  received  and  analyzed,  you  will  be  asked  to  review  and  revise  your  initial 
responses  to  questions  4  and  5  based  on  responses  provided  by  the  entire  group.  Subsequent  rounds  will  be 
announced  as  needed  and  all  research  will  conclude  by  6  April  2012. 

Background  questions: 

1.  Personal  Information: 

a.  Rank/Grade: 

b.  Current  Duty  Title: 

c.  Time  in  Current  Position: 

d.  (If  military)  Core  AFSC: 

2.  How  long  have  you  been  involved  with  building  partnerships  among  foreign  partner  nations? 

3.  In  what  capacity  have  you  dealt  with  foreign  nation  partnership  issues? 

Research  questions: 

Please  answer  the  following  questions  as  clearly  and  concisely  as  possible  without  omitting  critical 
information  required  for  the  group  to  consider  your  opinions.  Provide  any  appropriate  rationale  for 
your  responses. 

4.  What  minimum  criteria  (i.e.  performance  objectives,  key  indicators,  etc)  would  you  recommend  the 
MSAS  use  to  measure  their  effectiveness  at  accomplishing  their  Building  Partnership  Capacity  mission? 

5.  What  reporting  mechanism  would  you  recommend  the  MSAS  use  to  track  and  report  their  unit 

effectiveness? 
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Appendix  B 


Mission 


Air  Mobility  Command's  Building  Part¬ 
ner  Capacity  (BPC)  consists  of  tar¬ 
geted  efforts  to  improve  the  collective 
capabilities  and  performance  of  the 
DoD  and  its  partners.  Building  Part¬ 
ner  and  Building  Partner  Capacity  en¬ 
ables  partner  nations  to  build  afford¬ 
able  and  sustainable  mobility  systems 
of  their  own  that  are  interoperable 
with  AMC's  system. 


The  CRWs'  experience  in  emergency 
i espouse  with  utl»w  nations,  along  with 
their  inherent  expeditionary  skills,  make 
CRWs  the  premier  units  for  emerging 
AMC  BP  and  BPC  efforts.  The  end  state 
is  to  help  prevent  conflicts,  defeat  insur¬ 
gents  and  allow  timely  response  to  dis¬ 
asters  by  enabling  partner  nations  to 
create  a  self-sufficient  air  mobility  sys¬ 
tem  compatible  with  international  part- 


Mobility  Support 
Advisory  Squadrons 

Mobility  Support  Advisory  Squadrons  (MSAS) 
execute  Air  mobility  Command's  BPC  mission. 
The  571  MSAS,  located  at  Travis  Air  Force 
Base,  Calif,  is  aligned  with  and  supports  United 
States  Southern  Command.  The  818  MSAS, 
located  at  Joint  Base  McGuire-Dix-Lakehurst, 
N.J.,  is  aligned  with  and  supports  United  States 
Africa  Command.  Both  MSAS  units  are  expertly 
manned  and  equipped  to  work  alongside  part¬ 
ner  nation  military  personnel  in  developing  the 
six  core  competencies  of: 


Communications 

Effective  voice  and  data  infrastructure  supports 
command,  control,  communications  and  com¬ 
puter  functions  integral  to  a  nation's  ability  to 
reach  and  serve  its  citizens. 

Airfield  Operations 

Improving  airfield  and  aircraft  management 
leverages  limited  air  assets  by  optimizing 
ground  operations.  Through  improved  airfield 
operations,  partner  nations  can  create  airlift 


command  and  control,  airfield  operations,  com¬ 
munications,  aenal  port,  aircraft  maintenance, 
and  aeromedical  evacuation.  Each  MSAS  will  be 
comprised  of  more  than  70  technical  special¬ 
ists,  representing  nearly  25  Air  Force  career 
fields.  Initial  Operating  Capability  for  the 
squadrons  is  scheduled  to  be  achieved  Dec. 
2011,  while  Final  Operating  Capability  is  set 
for  Dec.  2012. 

Command  and  Control 

Command  and  Control  is  an  essential  tool  for  a 
nation  to  create  a  government  plan  and  exe¬ 
cute  it  with  flexibility  across  their  territory. 


capacity  without  acquiring  more  aircraft.  Im¬ 
provements  include:  airfield  operating  restric¬ 
tions,  en route  maintenance,  safety  and  air 
traffic  control  systems. 

Aerial  Support 

Developing  aerial  port  capabilities  improves 
terminal  operations,  aircraft  on  and  off-load 
procedures  and  efficiency  of  cargo  and  pas¬ 
senger  movement. 

Aircraft  Maintenance 

A  robust  aircraft  maintenance  program  is  criti¬ 
cal  to  sustainment  of  equipment  and  aerial  op¬ 
erations 
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Appendix  C 


Mobility  Support  Advisory  Squadron’s  BPC  Mission  Delphi  Study  Questionnaire  #2 

Thank  you  for  agreeing  to  participate  in  this  Delphi  Study.  The  purpose  of  this  research  is  to 
determine  what  appropriate  measurement  criteria  and  reporting  mechanism  the  Mobility  Support 
Advisory  Squadron  (MSAS)  could  use  to  track  and  report  unit  effectiveness  toward  their  Building 
Partnership  Capacity  mission.  The  sponsor  for  this  research  is  Colonel  Christopher  A.  Pike,  Vice 
Commander,  21st  Expeditionary  Mobility  Task  Force,  Joint  Base  McGuire-Dix-Lakehurst,  NJ. 

Please  note  the  following: 

Benefits  and  risks:  There  are  no  personal  benefits  or  risks  for  participating  in  this  study.  Your  participation 
in  completing  this  questionnaire  should  take  less  than  1 5  minutes  per  round. 

Confidentiality:  Your  responses  are  completely  confidential,  and  your  identity  will  remain  anonymous. 

No  individual  data  will  be  reported;  only  data  in  aggregate  will  be  made  public.  Data  will  be  kept  in  a 
secure,  locked  cabinet  to  which  only  the  researchers  will  have  access.  If  you  have  any  questions  or 
concerns  about  your  participation  in  this  study,  please  contact: 


JOSEPH  E.  WHITTINGTON,  JR.,  Major,  USAF 
IDE  Student,  Advanced  Study  of  Air  Mobility 
USAF  Expeditionary  Center 
JB  McGuire-Dix-Lakehurst,  NJ 
DSN  312-650-7748 
Comm  609-754-7748 


ALAN  R.  HEMINGER,  Ph.D. 

Associate  Professor  of  Management  Information 
Systems 

Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  OH 
Voice:  937-255-3636  (785-3636  DSN)  ext  7405 


Voluntary  consent:  Your  participation  is  completely  voluntary.  You  have  the  right  to  decline  to  answer 
any  question,  to  refuse  to  participate,  or  to  withdraw  at  any  time.  Your  decision  of  whether  or  not  to 
participate  will  not  result  in  any  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Completion 
of  the  questionnaire  implies  your  consent  to  participate. 

Purpose: 

This  qualitative  research  effort  seeks  to  determine  what  appropriate  measurement  criteria  and 
reporting  mechanism  the  MSAS  could  use  to  track  and  report  unit  effectiveness  toward  BPC.  By 

responding,  you  have  the  unique  opportunity  to  shape  how  the  MSAS  can  tie  their  tactical-level 
actions  to  strategic  outcomes.  The  data  obtained  through  this  study  will  form  recommendations  that  will 
be  offered  to  Air  Force  decision  makers  at  the  Headquarters,  Major  Command  (Air  Mobility  Command), 
and  wing  levels  in  order  to  shape  the  future  of  this  mission.  Thank  you  for  participating  in  this  study.  I 
appreciate  your  time  and  candid  responses. 

Process: 

1.  Please  complete  this  questionnaire  electronically  and  return  it  to:  joseph.whittington@us.af.mil  no 
later  than  2  March  2012.  If  you  have  questions,  I  can  be  reached  at  609-754-7748  or  via  DSN  650-7748. 

2.  This  questionnaire  is  an  instrument  of  a  Delphi  Study.  The  Delphi  method  is  an  iterative,  group 
communication  process  which  is  used  to  collect  and  distill  the  judgments  of  experts  using  a  series  of 
questionnaires  interspersed  with  feedback.  The  questionnaires  are  designed  to  focus  on  problems, 
opportunities,  solutions,  or  forecasts.  Each  questionnaire  is  developed  based  on  the  results  of  the  previous 
questionnaire.  The  process  continues  until  the  research  question  is  answered.  For  example,  when 
consensus  is  reached  or  sufficient  information  has  been  exchanged.  This  usually  takes,  on  average,  3-4 
rounds. 
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3.  This  is  Round  #2  of  the  study.  Once  all  questionnaire  responses  are  received  and  analyzed,  you  will  be 
asked  to  review  and  revise  your  initial  responses  based  on  collective  responses  provided  by  the  entire 
group.  Subsequent  rounds  will  be  announced  as  needed  and  all  research  will  conclude  by  6  April  2012. 


Results  from  Questionnaire  #1 

Below  are  the  responses  from  Questionnaire  #1.  Please  review  the  responses  and  then  continue  to 
Questionnaire  #2.  The  group’s  responses  were  categorized  into  7  main  subject  areas  for  the  first  research 
question  and  6  main  subject  areas  for  the  second  research  question.  The  responses  are  listed  by  the 
frequency  the  criteria  was  mentioned. 

Original  question:  What  minimum  criteria  (i.e.  performance  objectives,  key  indicators,  etc.)  would  you 
recommend  the  MSAS  use  to  measure  their  effectiveness  at  accomplishing  their  Building  Partnership 
Capacity  mission? 


Category 

Frequency 

Ability  to  gain/maintain  access  to  Partner  Nation  (PN) 

14 

Subcategory: 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to  information,  facilities, 
airfields,  equipment,  etc.) 

9 

Ability  to  establish  key  contacts 

4 

Ability  to  make  repetitive/persistent  visits  with  PN 

1 

Actions  tied  to  U.S.  National/Strategic  policy  goals 

8 

Subcategory: 

MSAS  engagements  are  in  support  of  U.S.  National  Security  policy,  Geographic 
Combatant  Commander  (GCC)  and  Component  Plans  (e.g.  Theater  Security 
Cooperation  Plans,  Campaign  Plans,  and  Air  Force  Component  Campaign  Support 
Plan)  and  Ambassador  Country  Plans  (e.g.  Mission  Strategic  and  Resource  Plan) 

6 

MSAS  actions  are  integrated  with  other  U.S.  Government  agency  actions 

2 

Post-engagement  outcomes 

6 

Subcategory: 

MSAS  contributed  towards  PN's  increased  capability  and  capacity  (e.g.  response 
during  crisis,  stable  or  increasing  mission  capability  rate,  sustained  personnel 
training/task  proficiency,  interoperability  with  U.S.)  as  reported  by  country  team, 

GCC  staff 

3 

Extent  to  which  MSAS  accomplished  policy  objectives  (e.g.  GCC  end  states  met) 

2 

Notable  improvement/development  of  key  relationships  with  PN  (e.g.  within 
government,  military) 

1 

Ability  to  further  develop  MSAS  capability 

6 

Subcategory: 

Ability  to  identify  and  recruit  personnel  for  MSAS  (e.g.  locate,  train,  develop, 
track/place)  and  seek  to  establish  a  formal  career  track  and  career 
progression/promotion  opportunities 

2 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g.  language/culture 
familiarization,  general  advising  skills,  etc.) 

2 

Ability  to  develop  internal  processes  that  support  MSAS  mission  (e.g.  record 
keeping  for  information  continuity) 

1 

Ability  to  develop  key  relationships  (e.g.  other  military  services,  Department  of 

State,  interagency  organizations,  regional  centers  like  the  Marshall  Center  in 

Europe) 

1 
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Ability  to  set  realistic  goals  for  PN  and  provide  the  tools/skills  to  attain  them 

4 

Subcategory: 

Ability  to  develop  plans,  milestones,  projects  the  PN  can  accomplish  within  realistic 
timeframe 

4 

Ability  to  develop  standards  of  performance 

4 

Subcategory: 

Ability  to  develop  achievable  Mission  Essential  Task  Lists  and  performance 
standards  for  PN 

4 

Number  of  taskings  received  vs.  number  of  taskings  completed 

2 

Subcategory: 

Requestor  and  types  of  requests  received  (e.g.  Assess,  Train,  Advise,  Assist,  Equip) 

1 

Frequency  and  duration/longevity  interactions  with  PN 

1 

Total 

44 

Original  question:  What  reporting  mechanism  would  you  recommend  the  MSAS  use  to  track  and  report 
their  unit  effectiveness? 


Category 

Frequency 

After  Action  Reports 

5 

Centralized  data  collection  and  dissemination  systems 

5 

Subcategory: 

Make  entries  into  GCC/Air  Force  Theater  Security  Cooperation  Management 
Information  System  (TSCMIS) 

4 

Make  entries  into  Joint  Lessons  Learned  Information  System  (JLLIS) 

1 

Spreadsheets 

5 

Subcategory: 

Track  PN  progress  following  MSAS  engagements  using  a  numeric  rating  scale  (e.g.  1 
-  5  scale:  (0)  -  No  capability  and  (5)  -  Can  complete  without  assistance 

5 

Feedback  from  Surveys 

3 

Subcategory: 

Survey  results  from  GCC,  Component  staff,  Country  team,  PN 

3 

Formal  reports 

3 

Subcategory: 

Submit  annual/semi-annual  reports  to  GCC,  Component  staff,  and  Air  Mobility 
Command  through  MSAS  chain  of  command 

3 

Unit  Readiness  reporting 

1 

Subcategory: 

Report  unit  mission  readiness  monthly  through  Status  of  Resources  and  Training 
System  (SORTS) 

1 

Total 

22 

Questionnaire  #2:  Rating  of  Criteria 

1)  Please  review  each  of  the  following  items  identified  in  Questionnaire  #1  as  minimum  criteria  the 
MSAS  could  use  to  measure  their  effectiveness  at  accomplishing  their  Building  Partnership  Capacity 
mission.  If  you  wish  to  add  comments  expressing  agreement,  disagreement,  or  clarification  concerning  an 
item,  please  do  so  in  the  space  provided.  Brevity  and  clarity  will  facilitate  analysis. 
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2)  Please  rate  the  following  items  by  level  of  importance  as  you  perceive  them  at  this  time. 


5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 


Criteria 

Rating 

Comments 

Ability  to  assess  Partner  Nation  (PN)  capability/capacity  (e.g.  access  to 
information,  facilities,  airfields,  equipment,  etc.) 

Ability  to  establish  key  contacts  within  PN  government,  military,  etc. 

Ability  to  make  repetitive/persistent  visits  with  PN 

MSAS  engagements  are  in  support  of  U.S.  National  Security  policy, 
Geographic  Combatant  Commander  (GCC)  and  Component  Plans  (e.g. 
Theater  Security  Cooperation  Plans,  Campaign  Plans,  and  Air  Force 
Component  Campaign  Support  Plan)  and  Ambassador  Country  Plans  (e.g. 
Mission  Strategic  and  Resource  Plan) 

MSAS  actions  are  integrated  with  other  U.S.  Government  agency  actions 
(e.g.  State  Department,  U.S.  Agency  for  International  Development,  other 
military  units) 

MSAS  contributed  towards  PN's  increased  capability  and  capacity  (e.g. 
response  during  crisis,  stable  or  increasing  mission  capability  rate, 
sustained  personnel  training/task  proficiency,  interoperability  with  U.S.)  as 
reported  by  country  team,  GCC  staff 

Extent  to  which  MSAS  accomplished  policy  objectives  (e.g.  GCC  end  states 
met) 

Notable  improvement/development  of  key  relationships  with  PN  (e.g. 
within  government,  military) 

Ability  to  identify  and  recruit  personnel  for  MSAS  (e.g.  locate,  train, 
develop,  track/place)  and  seek  to  establish  a  formal  career  track  and 
career  progression/promotion  opportunities 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g. 
language/culture  familiarization,  general  advising  skills,  etc) 

Ability  to  develop  internal  processes  that  support  MSAS  mission  (e.g. 
record  keeping  for  information  continuity) 

Ability  to  develop  key  relationships  (e.g.  other  military  services, 

Department  of  State,  interagency  organizations,  regional  centers  like  the 
Marshall  Center  in  Europe) 

Ability  to  develop  plans,  milestones,  projects  the  PN  can  accomplish  within 
realistic  timeframe 

Ability  to  develop  achievable  Mission  Essential  Task  Lists  and  performance 
standards  for  PN 

Requestor  and  types  of  requests  received  (e.g.  Assess,  Train,  Advise, 

Assist,  Equip) 

Frequency  and  duration/longevity  of  interactions  with  PN 
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Questionnaire  #2:  Rank  Order  Criteria  (continued) 


1)  Please  review  each  of  the  following  items  identified  in  Questionnaire  #1  as  the  reporting  mechanism 
the  MSAS  could  use  to  track  and  report  their  unit  effectiveness.  If  you  wish  to  add  comments  expressing 
agreement,  disagreement,  or  clarification  concerning  an  item,  please  do  so  in  the  space  provided.  Brevity 
and  clarity  will  facilitate  analysis. 

2)  Please  rate  the  following  items  by  level  of  importance  as  you  perceive  them  at  this  time. 


5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 


Criteria 

Rating 

Comments 

After  Action  Reports 

Make  entries  into  Geographic  Combatant  Commander  (GCC)/Air  Force 
Theater  Security  Cooperation  Management  Information  System  (TSCMIS) 

Make  entries  into  Joint  Lessons  Learned  Information  System  (JLLIS) 

Track  Partner  Nation  (PN)  progress  following  MSAS  engagements  using  a 
spreadsheet  and  numeric  rating  scale  (e.g.  1  -  5  scale:  (0)  -  No  capability 
and  (5)  -  Can  complete  without  assistance) 

Survey  results  from  GCC,  Component  staff  Country  team,  PN 

Submit  annual/semi-annual  reports  to  GCC,  Component  staff,  and  Air 
Mobility  Command  through  MSAS  chain  of  command 

Report  unit  mission  readiness  monthly  through  Status  of  Resources  and 
Training  System  (SORTS) 
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Appendix  D 


Mobility  Support  Advisory  Squadron’s  BPC  Mission  Delphi  Study  Questionnaire  #3 

Thank  you  for  agreeing  to  participate  in  this  Delphi  Study.  The  purpose  of  this  research  is  to 
determine  what  appropriate  measurement  criteria  and  reporting  mechanism  the  Mobility  Support 
Advisory  Squadron  (MSAS)  could  use  to  track  and  report  unit  effectiveness  toward  their  Building 
Partnership  Capacity  mission.  The  sponsor  for  this  research  is  Colonel  Christopher  A.  Pike,  Vice 
Commander,  21st  Expeditionary  Mobility  Task  Force,  Joint  Base  McGuire-Dix-Lakehurst,  NJ. 

Please  note  the  following: 

Benefits  and  risks:  There  are  no  personal  benefits  or  risks  for  participating  in  this  study.  Your 
participation  in  completing  this  questionnaire  should  take  less  than  15  minutes  per  round. 

Confidentiality:  Y our  responses  are  completely  confidential,  and  your  identity  will  remain  anonymous. 
No  individual  data  will  be  reported;  only  data  in  aggregate  will  be  made  public.  Data  will  be  kept  in  a 
secure,  locked  cabinet  to  which  only  the  researchers  will  have  access.  If  you  have  any  questions  or 
concerns  about  your  participation  in  this  study,  please  contact: 


JOSEPH  E.  WHITTINGTON,  JR.,  Major,  USAF 
IDE  Student,  Advanced  Study  of  Air  Mobility 
USAF  Expeditionary  Center 
JB  McGuire-Dix-Lakehurst,  NJ 
DSN  312-650-7748 
Comm  609-754-7748 


ALAN  R.  HEMINGER,  Ph.D. 

Associate  Professor  of  Management  Information 
Systems 

Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  OH 
Voice:  937-255-3636  (785-3636  DSN)  ext  7405 


Voluntary  consent:  Your  participation  is  completely  voluntary.  You  have  the  right  to  decline  to  answer 
any  question,  to  refuse  to  participate,  or  to  withdraw  at  any  time.  Your  decision  of  whether  or  not  to 
participate  will  not  result  in  any  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled. 
Completion  of  the  questionnaire  implies  your  consent  to  participate. 


Purpose: 

This  qualitative  research  effort  seeks  to  determine  what  appropriate  measurement  criteria  and  reporting 
mechanism  the  MSAS  could  use  to  track  and  report  unit  effectiveness  toward  BPC.  By  responding,  you 

have  the  unique  opportunity  to  shape  how  the  MSAS  can  tie  their  tactical-level  actions  to  strategic 
outcomes.  The  data  obtained  through  this  study  will  form  recommendations  that  will  be  offered  to  Air 
Force  decision  makers  at  the  Headquarters,  Major  Command  (Air  Mobility  Command),  and  wing  levels 
in  order  to  shape  the  future  of  this  mission.  Thank  you  for  participating  in  this  study.  I  appreciate  your 
time  and  candid  responses. 

Process: 

1.  Please  complete  this  questionnaire  electronically  and  return  it  to:  joseph.whittington@us.af.mil  no 
later  than  23  March  2012.  If  you  have  questions,  I  can  be  reached  at  609-754-7748  or  via  DSN  650- 
7748. 


2.  This  questionnaire  is  an  instrument  of  a  Delphi  Study.  The  Delphi  method  is  an  iterative,  group 
communication  process  which  is  used  to  collect  and  distill  the  judgments  of  experts  using  a  series  of 
questionnaires  interspersed  with  feedback.  The  questionnaires  are  designed  to  focus  on  problems, 
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opportunities,  solutions,  or  forecasts.  Each  questionnaire  is  developed  based  on  the  results  of  the 
previous  questionnaire.  The  process  continues  until  the  research  question  is  answered.  For  example, 
when  consensus  is  reached  or  sufficient  information  has  been  exchanged.  This  usually  takes,  on 
average,  3-4  rounds. 

3.  This  is  the  3rd  and  Final  Round  of  the  study.  Once  all  questionnaire  responses  are  received,  an 
analysis  of  the  Round  3  will  be  conducted  and  the  results  will  be  summarized  and  sent  to  you  in  a  final 
report. 


Part  1  -  Results  from  Questionnaire  #2 

Below  are  the  results  from  the  previous  questionnaire.  Based  on  the  group’s  ratings,  consensus  was 
reached  on  9  criteria  for  the  first  research  question  and  4  criteria  for  the  second  research  question. 

The  criteria  below  were  listed  in  order  based  on  the  arithmetic  median  and  interquartile  range  (the  range  that 
contains  the  answers  of  the  middle  50  percent  of  the  respondents)  of  the  group’s  responses.  For  this 
study,  an  interquartile  range  (IQR)  of  1  or  less  is  an  indicator  of  consensus. 

Task:  First,  review  the  ratings  in  left-hand  column  for  both  sections.  Then,  select  “Yes”  or  “No”  in 
the  box  below.  If  you  select  “Yes,”  then  you  are  finished  with  this  section.  If  you  select  “No,”  please  re¬ 
rank  and  comment  as  needed.  Be  sure  to  use  the  same  5-point  rating  scale  from  the  last  round.  Once 
completed,  continue  on  to  Part  2  -  Questionnaire  3. 

I  agree  with  ratings  as  determined  by  the  group  in  Round  2: 


Yes  (Proceed  to  part  2) 

No  (Re-rank,  comment,  then  proceed  to  Part  2) 

Section  1 :  The  minimum  criteria  the  MSAS  could  use  to  measure  their  effectiveness  at  accomplishing 
their  Building  Partnership  Capacity  mission. 


5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Litt 

e  Importance,  l=Unimportant 

Your 

Rating 

Group's 

Rating/IQR 

Criteria 

New  Rating/Commentfs) 

5 

5/1 

MSAS  contributed  towards  Partner  Nation's  (PN)  increased 
capability  and  capacity  (e.g.  response  during  crisis,  stable 
or  increasing  mission  capability  rate,  sustained  personnel 
training/task  proficiency,  interoperability  with  U.S.)  as 
reported  by  country  team,  GCC  staff 

5 

5/1 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g. 
language/culture  familiarization,  general  advising  skills, 
etc) 

5 

5/1 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to 
information,  facilities,  airfields,  equipment,  etc.) 
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4 

5/1 

MSAS  engagements  are  in  support  of  U.S.  National  Security 
policy,  Geographic  Combatant  Commander  (GCC)  and 
Component  Plans  (e.g.  Theater  Security  Cooperation  Plans, 
Campaign  Plans,  and  Air  Force  Component  Campaign 
Support  Plan)  and  Ambassador  Country  Plans  (e.g.  Mission 
Strategic  and  Resource  Plan) 

4 

4/0.25 

Ability  to  develop  internal  processes  that  support  MSAS 
mission  (e.g.  record  keeping  for  information  continuity) 

4 

4/1 

Ability  to  develop  achievable  Mission  Essential  Task  Lists 
and  performance  standards  for  PN 

4 

4/1 

Requestor  and  types  of  requests  received  (e.g.  Assess, 

Train,  Advise,  Assist,  Equip) 

4 

4/1 

Frequency  and  duration/longevity  of  interactions  with  PN 

4 

4/1 

Ability  to  develop  key  relationships  (e.g.  other  military 
services,  Department  of  State,  interagency  organizations, 
regional  centers  like  the  Marshall  Center  in  Europe) 

Section  2:  Reporting  mechanism  the  MSAS  could  use  to  track  and  report  their  unit  effectiveness: 


5  =Very  Important,  4=Important,  3=Moderately  Important,  2~ Of  Little  Importance,  l=Unimportant 


Your 

Rating 

Group's 

Rating/IQR 

Criteria 

New  Rating/Comment(s) 

5 

5/1 

After  Action  Reports 

4 

4/1 

Survey  results  from  GCC,  Component  staff,  Country  team, 

PN 

5 

3/0.5 

Report  unit  mission  readiness  monthly  through  Status  of 
Resources  and  Training  System  (SORTS) 

4 

3/1 

Make  entries  into  Joint  Lessons  Learned  Information 

System  (JLLIS) 

Part  2  -  Questionnaire  #3:  Reviewing  and  Re-rating  the  Remaining  Criteria 

The  items  below  did  not  reach  consensus  in  the  last  round.  Therefore,  items  in  section  1  and  2  below 
should  be  reviewed  and  re-assessed  to  see  if  the  lack  of  consensus  still  holds.  If  you  do  not  find  yourself 
agreeing  with  the  majority  opinion,  please  explain  your  reasoning  to  help  us  better  understand  the 
divergence  of  opinion.  The  criteria  were  analyzed  using  the  arithmetic  median  of  the  responses  given  in 
Questionnaire  #2. 

Task:  Please  re-rate,  as  required,  the  items  in  both  sections  by  level  of  importance  as  you  perceive 
them  at  this  time  using  the  5-point  rating  scale  below.  Add  brief,  concise  comments  for  clarification.  If 
you  want  to  keep  your  original  rating,  please  add  “No  Change”  to  the  comments  section. 

Section  1:  The  minimum  criteria  the  MSAS  could  use  to  measure  their  effectiveness  at  accomplishing 
their  Building  Partnership  Capacity  mission.  There  are  7  items  included. 

5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 
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Your 

Rating 

Group’s 

Rating 

from 

Round  2 

Criteria 

Group's  Comments 

New  Rating/Comment(s) 

5 

5 

Ability  to  establish  key  contacts 
within  PN  government,  military, 

etc. 

-  Relationships  are  #1  priority. 

-  This  does  not  tell  me  how 

effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

-  No  substantial  contacts,  no 
mission. 

-  Dept  of  State  responsibility. 

-  Ways  (Ends,  Ways,  Means). 

4 

4 

Ability  to  make 

repetitive/persistent  visits  with  PN 

-  Key  to  MSAS  mission. 

-  Means  (Ends,  Ways,  Means). 

-  This  does  not  tell  me  how 

effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

-  Establishes/maintains 
relationships  and  results  in 
minimal  re-learning  of 
requirements/expectations. 

-  This  criterion  depends  entirely 
on  the  nature  of  why  return  visit 
is  warranted. 

-  Depends  on  what  you  want  to 
do  with  the  country. 

-To  build  a  lasting  capability, 
regular  engagement  a  must. 

4 

4 

MSAS  actions  are  integrated  with 
other  U.S.  Government  agency 
actions  (e.g.  State  Department, 

U.S.  Agency  for  International 
Development,  other  military  units) 

-  If  you  aren't  Interagency  and 
Joint,  you  won't  be  successful. 

-  Very  important  if  there  are 
other  engagements  in  progress 
but  not  required  to  move 
forward. 

-  Ends  (Ends,  Ways,  Means). 

-  Important  to  long-term 
development. 

-  This  does  not  tell  me  how 

effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

4 

4 

Extent  to  which  MSAS 

accomplished  policy  objectives 
(e.g.  GCC  end  states  met) 

-  This  is  the  one  criterion  that  all 

others  fall  under. 

-  Difficult  to  measure  since 
capacity  increases  will  probably 
take  extended  periods  of  time 
that  go  beyond  one  group  of 
MSAS  advisor's  involvement. 

-  Policies  can  conflict  and  be 
interpreted  differently. 

-  Ends  (Ends,  Ways,  Means). 

-  Very  important  if  policy 
objectives  are  realistic;  focus 
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5 

4 

Notable 

improvement/development  of  key 
relationships  with  PN  (e.g.  within 
government,  military) 

-  How  do  you  know  when  you've 
attained  it. ..does  it  mean  you 
can  stop? 

-  This  would  be  a  specific  policy 
or  military  objective  the  MSAS 
to  accomplish;  one  could 
succeed  at  everything  else  and 
still  fail  at  this  objective. 

-  Ways  (Ends,  Ways,  Means). 

-  Along  with  actual  capability 
increases,  the  relationships  are 
right  there  with  importance. 

-  If  you  don't  have  relationships, 
you  have  nothing. 

-  Very  hard  to  measure,  but 
may  be  only  thing  to  measure 
early  on. 

4 

4 

Ability  to  identify  and  recruit 
personnel  for  MSAS  (e.g.  locate, 
train,  develop,  track/place)  and 

seek  to  establish  a  formal  career 

track  and  career 

progression/promotion 

opportunities 

-  This  can't  be  understated; 
need  to  build  a  professional, 
appealing,  and  career  enhancing 
career  field;  if  not  we  won't 
succeed. 

-  Ability  to  identify  and  recruit  is 
a  5  but  career  track  is  a  1. 

-  This  does  not  tell  me  how 

effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

-  Means  (Ends,  Ways,  Means). 

-  Important  to  get  the  right 
people  in  the  unit  and  get  the 
right  jobs  after  the  assignment. 

5 

4 

Ability  to  develop  plans, 
milestones,  projects  the  PN  can 
accomplish  within  realistic 

timeframe 

-  A  critical  step. 

-  Need  to  build  milestones 
based  on  PN  capabilities  not 

USAF  standards. 

-  This  one  is  a  minefield  since 
dependent  on  factors  outside 

DoD  control;  advisors  should  be 
assessed  on  ability  to  develop 
relationships. 

-  Means  (Ends,  Ways,  Means). 

-  This  does  not  tell  me  how 
effective  1  am  or  am  not.  It  is 
important  but  does  not  speak  to 
the  question. 

-  Timelines  are  secondary. 

-  Initial  milestones  may  be 
faulty.. .but  as  they're  made  and 
adjusted  they  provide  objectives 
to  be  made/reported. 
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Section  2:  Reporting  mechanism  the  MSAS  could  use  to  track  and  report  their  unit  effectiveness. 
There  are  3  items  included. 


5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 


Your 

Rating 

Group’s 

Rating 

from 

Round  2 

Criteria 

Group's  Comments 

New  Rating/Comment(s) 

5 

4 

Submit  annual/semi-annual 
reports  to  GCC,  Component  staff, 
and  Air  Mobility  Command 
through  MSAS  chain  of  command 

-  Excellent  idea  and  critical  that 

non-advisors  understand  what 
you  are  accomplishing  and 
dollar  cost  benefit. 

-  Linked  to  measures  of  effect 
identified  in  specific  Theater 
Campaign  plans  down  to 
country  level. 

-  If  MSAS  is  going  to  stay  in 
business,  they  are  going  to  have 
to  communicate  effectively  with 
their  customers  and  bosses. 

3 

4 

Track  Partner  Nation  progress 
following  MSAS  engagements 
using  a  spreadsheet  and  numeric 
rating  scale  (e.g.  1  -  S  scale:  (0)  - 
No  capability  and  (5)  -  Can 
complete  without  assistance) 

-  Very  Important. 

-  Looks  good  on  PowerPoint  and 
metrics  are  good,  but  reality  is 
more  detailed  than  this. 

-  Not  sure  this  is  the  best 
evaluation  suggestion. 

-  Very  important  but  should  be 
close  hold  due  to  possible  PN 
embarrassment;  critical  to  the 
constant  assessment  process. 

-  Very  hard  to  measure,  but 
may  be  only  thing  to  measure 
early  on. 

5 

3 

Make  entries  into  Geographic 
Combatant  Commander  (GCC)/Air 
Force  Theater  Security 

Cooperation  Management 
Information  System  (TSCMIS) 

-  Instrumental  to  validating  and 
justifying  MSAS. 
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Appendix  E 


Mobility  Support  Advisory  Squadron’s  BPC  Mission  Delphi  Study  Final  Report 

Thank  you  for  participating  in  this  Delphi  study.  The  purpose  of  this  research  is  to  determine 
what  appropriate  measurement  criteria  and  reporting  mechanism  the  Mobility  Support  Advisory 
Squadron  (MSAS)  could  use  to  track  and  report  unit  effectiveness  toward  their  Building  Partnership 
Capacity  mission.  The  sponsor  for  this  research  is  Colonel  Christopher  A.  Pike,  former  Vice 
Commander,  21st  Expeditionary  Mobility  Task  Force,  Joint  Base  McGuire-Dix-Lakehurst,  NJ. 

Please  note  the  following: 

Benefits  and  risks:  There  are  no  personal  benefits  or  risks  for  participating  in  this  study. 

Confidentiality:  Your  responses  are  completely  confidential,  and  your  identity  will  remain  anonymous. 
No  individual  data  will  be  reported;  only  data  in  aggregate  will  be  made  public.  Data  will  be  kept  in  a 
secure,  locked  cabinet  to  which  only  the  researchers  will  have  access.  If  you  have  any  questions  or 
concerns  about  your  participation  in  this  study,  please  contact: 


JOSEPH  E.  WHITTINGTON,  JR.,  Major,  USAF 
IDE  Student,  Advanced  Study  of  Air  Mobility 
USAF  Expeditionary  Center 
JB  McGuire-Dix-Lakehurst,  NJ 
DSN  312-650-7748 
Comm  609-754-7748 


ALAN  R.  HEMINGER,  Ph.D. 

Associate  Professor  of  Management  Information 
Systems 

Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  OH 
Voice:  937-255-3636  (785-3636  DSN)  ext  7405 


Process: 

The  questionnaires  you  completed  were  instruments  of  a  Delphi  study.  The  Delphi  method  is  an 
iterative,  group  communication  process  used  to  collect  and  distill  the  judgments  of  experts  using  a  series 
of  questionnaires  interspersed  with  feedback.  Each  questionnaire  was  developed  based  on  the  results  of 
the  previous  questionnaire.  The  process  continued  until  the  research  questions  were  answered.  Three 
rounds  were  conducted,  and  a  total  of  3  questionnaires  were  used  for  this  study. 


Part  1  -  Results  from  Questionnaire  #3 

Below  are  the  results  from  the  final  questionnaire.  After  reviewing  the  ratings  and  comments  from 
round  2,  the  group  re-evaluated  their  ratings  and  reached  consensus  on  15  out  of  the  16  criteria  for  the 
first  research  question — an  increase  from  9  items  in  round  2.  Consensus  was  reached  for  6  out  of  the 
7  criteria  for  the  second  research  question — an  increase  from  4  items  in  round  2.  A  total  of  76  rating 
changes  were  made  for  the  first  research  question,  and  38  rating  changes  were  made  for  the  second 
research  question.  1 1  panel  members  agreed  with  the  collective  group  ratings  from  round  2.  Finally,  the 
overall  number  of  panel  member  comments  provided  was  noticeably  higher  in  round  3  than  in  round  2. 
12  panel  members  made  comments  in  questionnaire  2,  and  18  made  comments  in  questionnaire  3 — a 
50%  increase. 


The  criteria  below  are  listed  in  order  based  on  the  arithmetic  median  and  interquartile  range.  This  range 
contains  the  answers  from  the  middle  50  percent  of  the  respondents.  For  this  study,  an  interquartile  range 
(IQR)  of  1  or  less  is  an  indicator  of  consensus. 
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Section  1:  The  minimum  criteria  the  MSAS  could  use  to  measure  their  effectiveness  at  accomplishing 
their  Building  Partnership  Capacity  mission. 

5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  1  =Unimportant 


Criteria 

Group's 

Rating/IQR 

Criteria 

1 

5/0 

MSAS  contributed  towards  Partner  Nation's  (PN)  increased  capability  and  capacity  (e.g. 
response  during  crisis,  stable  or  increasing  mission  capability  rate,  sustained  personnel 
training/task  proficiency,  interoperability  with  U.S.)  as  reported  by  country  team,  GCC  staff 

2 

5/0 

MSAS  engagements  are  in  support  of  U.S.  National  Security  policy,  Geographic  Combatant 
Commander  (GCC)  and  Component  Plans  (e.g.  Theater  Security  Cooperation  Plans, 

Campaign  Plans,  and  Air  Force  Component  Campaign  Support  Plan)  and  Ambassador 

Country  Plans  (e.g.  Mission  Strategic  and  Resource  Plan) 

3 

5/0 

Ability  to  assess  PN  capability/capacity  (e.g.  access  to  information,  facilities,  airfields, 
equipment,  etc.) 

4 

5/0.5 

Ability  to  establish  key  contacts  within  PN  government,  military,  etc. 

5 

5/1 

Ability  to  develop  a  training  plan  for  MSAS  personnel  (e.g.  language/culture  familiarization, 
general  advising  skills,  etc) 

6 

4/0 

MSAS  actions  are  integrated  with  other  U.S.  Government  agency  actions  (e.g.  State 
Department,  U.S.  Agency  for  International  Development,  other  military  units) 

7 

4/0 

Frequency  and  duration/longevity  of  interactions  with  PN 

8 

4/0 

Requestor  and  types  of  requests  received  (e.g.  Assess,  Train,  Advise,  Assist,  Equip) 

9 

4/0 

Ability  to  develop  internal  processes  that  support  MSAS  mission  (e.g.  record  keeping  for 
information  continuity) 

10 

4/0 

Ability  to  develop  key  relationships  (e.g.  other  military  services,  Department  of  State, 
interagency  organizations,  regional  centers  like  the  Marshall  Center  in  Europe) 

11 

4/0.25 

Notable  improvement/development  of  key  relationships  with  PN  (e.g.  within  government, 
military) 

12 

4/0.25 

Ability  to  develop  achievable  Mission  Essential  Task  Lists  and  performance  standards  for  PN 

13 

4/0.5 

Ability  to  develop  plans,  milestones,  projects  the  PN  can  accomplish  within  realistic 

timeframe 

14 

4/1 

Extent  to  which  MSAS  accomplished  policy  objectives  (e.g.  GCC  end  states  met) 

15 

4/1 

Ability  to  make  repetitive/persistent  visits  with  PN 

Section  2:  Reporting  mechanism  the  MSAS  could  use  to  track  and  report  their  unit  effectiveness: 

5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  1  =Unimportant 


Criteria 

Group's 

Rating/IQR 

Criteria 

1 

5/0 

After  Action  Reports 

2 

4/0 

Survey  results  from  GCC,  Component  staff,  Country  team,  PN 

3 

4/1 

Track  Partner  Nation  progress  following  MSAS  engagements  using  a  spreadsheet  and 
numeric  rating  scale  (e.g.  1-5  scale:  (0)  -  No  capability  and  (5)  -  Can  complete  without 
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3 

4/1 

Track  Partner  Nation  progress  following  MSAS  engagements  using  a  spreadsheet  and 
numeric  rating  scale  (e.g.  1-5  scale:  (0)  -  No  capability  and  (5)  -  Can  complete  without 
assistance) 

4 

3/0 

Make  entries  into  Joint  Lessons  Learned  Information  System  (JLLIS) 

5 

3/0 

Report  unit  mission  readiness  monthly  through  Status  of  Resources  and  Training  System 
(SORTS) 

6 

3/1 

Make  entries  into  Geographic  Combatant  Commander  (GCC)/Air  Force  Theater  Security 
Cooperation  Management  Information  System  (TSCMIS) 

Section  3:  Items  that  group  consensus  was  not  reached  on: 

5  =Very  Important,  4=Important,  3=Moderately  Important,  2=Of  Little  Importance,  l=Unimportant 


Criteria 

Group's 

Rating/IQR 

Criteria 

Research 

Question 

1 

4/1.25 

Ability  to  identify  and  recruit  personnel  for  MSAS  (e.g.  locate,  train,  develop,  track/place) 
and  seek  to  establish  a  formal  career  track  and  career  progression/promotion 
opportunities 

Research 

Question 

2 

4/1.25 

Submit  annual/semi-annual  reports  to  GCC,  Component  staff,  and  Air  Mobility  Command 
through  MSAS  chain  of  command 

Part  2  -  Items  of  Importance 

Of  the  16  total  criteria  identified  as  minimum  criteria  the  MSAS  could  use  to  measure  their  effectiveness 
at  accomplishing  their  BPC  mission,  three  emerged  as  decidedly  important  in  that  they  attained  the 

highest  possible  median  score  of  5.0  and  greatest  degree  of  group  consensus  with  an  IQR  of  0.0: 

1.  MSAS  contributed  towards  Partner  Nation’s  (PN)  increased  capability  and  capacity  (e.g.  response 
during  crisis,  stable  or  increasing  mission  capability  rate,  sustained  personnel  training/task  proficiency, 
interoperability  with  U.S.)  as  reported  by  country  team,  GCC  staff; 

2.  MSAS  engagements  are  in  support  of  U.S.  National  Security  policy,  Geographic  Combatant 
Commander  (GCC)  and  Component  Plans  (e.g.  Theater  Security  Cooperation  Plans,  Campaign  Plans, 
and  Air  Force  Component  Campaign  Support  Plan)  and  Ambassador  Country  Plans  (e.g.  Mission 
Strategic  and  Resource  Plan); 

3.  Ability  to  assess  PN  capability/capacity  (e.g.  access  to  information,  facilities,  airfields,  equipment, 
etc.). 

Of  the  7  total  criteria  identified  as  the  reporting  mechanism  the  MSAS  could  use  to  track  and  report  their 
unit  effectiveness,  one  emerged  as  decidedly  important  in  that  it  attained  the  highest  possible  median 
score  of  5.0  and  greatest  degree  of  group  consensus  with  an  IQR  of  0.0: 

1 .  After  Action  Reports. 
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Part  3  -  Summary  of  Participation 


The  group’s  participation  throughout  the  study  was  phenomenal.  Researchers  have  reported  that 
participation  rates  from  40  to  75%  can  be  anticipated  with  the  Delphi  study.  This  study  concluded  with 
an  overall  98.3%  participation  rate.  Below  is  a  summary: 


Round 

#  Invited  to  Participate 

#  Participated 

Participation  Rate  (%) 

1 

20 

19 

95 

2 

20 

20 

100 

3 

20 

20 

100 

Overall  Participation 

98.3 

By  participating  in  this  study,  you  sought  the  unique  opportunity  to  shape  how  the  MSAS  can  tie 
their  tactical-level  actions  to  strategic  outcomes.  The  data  the  group  provided  through  this  study  will 
form  recommendations  that  will  be  offered  to  Air  Force  decision  makers  at  the  Headquarters,  Major 
Command  (Air  Mobility  Command),  and  wing  levels  in  order  to  shape  the  future  of  this  mission. 

Thank  you  again  for  participating  in  this  study.  I  greatly  appreciate  your  time  and  candid  responses. 

Very  Respectfully, 

JOSEPH  E.  WHITTINGTON,  JR.,  Major,  USAF 
Student,  Advanced  Study  of  Air  Mobility 
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Determining  Mobility  Support  Advisory 
Squadron  Effectiveness  In  Support  of 
Building  Partner  Capacity 


The  USAF  has  taken  the  leadership  role 
to  develop  the  aviation  capacity  and 
capability  of  partner  nations  (PN).  The 
key  to  accomplishing  this  objective  are 
Air  Advisors  (AA).  AAs  are  trained  and 
educated  to  assess,  advise,  train,  assist, 
and  equip  selected  PNs.  Air  Mobility 
Command  has  activated  two  Mobility 
Support  Advisory  Squadrons  (MSAS)  to 
manage  the  AAs  and  their  taskings. 
Initial,  broad-based  guidance  was 
provided  but  no  direction  was  given  on 
how  to  measure  and  report  MSAS 
effectiveness  at  conducting  their 
Building  Partner  Capacity  (BPC)  mission. 
This  research  utilized  a  Delphi  study  to 
determine  criteria  the  MSAS  could  use  to 
measure,  track  and  report  unit 
effectiveness  toward  accomplishing  their 
mission. 


Research  Goal 

■  Determine  appropriate  measurement 
criteria  the  MSAS  could  use  to  report 
unit  effectiveness  toward  BPC 
(Research  Question  (RQ)1) 

■  Determine  appropriate  reporting 
mechanism  the  MSAS  could  use  to 
report  unit  effectiveness  toward  BPC 
(Research  Question  (RQ)  2) 

iVk  f  air  rows  Demure  (fjmk 


Maj  Joseph  E.  Whittington,  Jr. 
Advisor:  Alan  R.  Heminger,  Ph.D. 
Sponsor:  Col  Christopher  A.  Pike 

Advanced  Study  of  Air  Mobility  (ASAM) 
Air  Force  Institute  of  Technology 


“The  BP  mission  of  AMC  is  to  conduct  operations  that  will  train, 
advise,  and  assist  PNs  in  the  development  of  an  Air  Mobility  System. ' 

■  Air  Mobility  Systems  Building  Partnerships 
Concept  of  Employment  2010 


S  =Very  Important  4=lmportwL  3=Moderately  Important  2=Of  Little  Importance.  l=Unlmportant 


■  would  you  rocommond  tt 
mpllshlng  their  Building  Part 


i  M£A£  uso  to  moacure  their 
ershtp  Capacity  mission? 


MSAS  contributed  toward*  Partner  Nation'*  (PN)  Increased  capability  and 
capacity  (e  g  response  during  crisis,  stable  or  increasing  mission  capability  ra 
sustained  personnel  tralnlng/tosk  proficiency.  Interoperability  with  U.S.)  as 
reported  by  country  team,  GCC  staff 


MSAS  engagements  are  in  support  erf  U.S.  National  Security  policy,  Geagraph 

Combatant  Commander  (GCC)  and  Component  Plans  <#.g.  Theater  security 

Cooperation  Plans,  Campaign  Plans,  and  Air  Force  Component  Campaign 

Support  Plan)  and  Ambassador  Country  Plans  (e.g.  Mission  Strategic  arid 
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information,  facilities. 


After  Action  Reports 


Final  Medlar 


Motivation 

The  tactical-level  actions  each  MSAS 
takes  with  their  assigned  PNs  must 
be  conveyed  in  some  manner  that 
demonstrates  effectiveness  and 
progress  of  the  overall  engagement. 


Application  -  Decision  Points 

■  20  experts  formed  the  Delphi  panel 

■  3  rounds  w/questionnaires 

■  66  initial  criteria  narrowed  to  23 

■  Consensus  reached  on  21  criteria 
•  15/16  for  RQ1  &  6/7  for  RQ2 

4  w/best  scores  (5-Median/O-IQR) 

■  Overall  98%  panel  response  rate 


Impacts/Contributions 

■  Delphi  panel  determined  initial 
criteria  the  MSAS  could  use  to 
measure/track/report  effectiveness 

■  AF  decision  makers  at  tactical, 
operational,  and  strategic  levels 
could  use  this  data  to  shape  the 
future  of  the  MSAS  &  BPC  efforts 

■  Data  from  this  study  can  help  the 
development  of  a  formal  AF 
assessment  framework 


Collaboration 

SAF/IA,  HAF  A3,  AMC  A3,  AETC  A3  & 
AAA,  AFSOC  A8  &  AFSOS,  CRWs 
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